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1.  INTRODUCTION 


1.1  Summary 


This  report  summarizes  a  three-year  program  of  research  and  development 
directed  toward  analysis  of  information  needs  and  real-time  handling  of 
information  selection  and  display  in  supervisory  control  of  advanced 
aircraft.  Its  purpose  is  to  establish  new  techniques  for  the  selection 
of  the  immediate,  essential  information  in  airborne  tactical  operations. 
The  techniques  center  on  the  use  of  multi -attribute  utility  models, 
adaptive  estimation  methods,  and  queueing  theory  formulations  to  model 
an  individual's  multi-task  supervisory  control  behavior. 

Specific  objectives  of  the  three-year  program  included  the  following: 

(1)  Formulate  a  working  taxonomy  of  supervisory  control  func¬ 
tions  in  advanced  aircraft  operations.  Relate  types  of 
computer-based  aiding  to  classes  of  supervisory  control 
functions. 

(2)  Develop  aiding  programs  for  continuous  information  monitor¬ 
ing  and  control  in  advanced  aircraft  operations.  The  pro¬ 
grams  take  into  account  the  immediate  needs  of  the  opera¬ 
tor  and  the  impact  on  other  unattended  processes. 

(3)  Investigate  experimentally  the  performance  and  domain  ap¬ 
plication  of  the  continuous  information  monitoring  and  con¬ 
trol  programs. 
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(4)  Investigate  the  behavioral  issues  of  operator  acceptance 
and  confidence  with  the  varied  forms  of  model -based  aiding. 

(5)  Develop  explicit  rules,  in  the  form  of  a  knowledge-based 
system,  for  controlling  transitions  between  aiding  forms. 

(6)  Produce  guidelines  for  field  application  of  the  information 
evaluation  and  management  programs  in  operational  airborne 
systems. 

The  initial  year's  work  established  techniques  for  real-time  information 
evaluation  and  selection  based  on  an  application  of  an  adaptive  multi¬ 
attribute  utility  (MAU)  model  and  queueing  theory  formulation.  The  com¬ 
bined  MAU/queueing  model  was  tested  in  a  Monte-Carlo  simulation,  and  the 
information  management  concept  based  on  the  proposed  model  was  superior 
to  those  based  on  traditional  priority  assignment.  The  study  also  sug¬ 
gested  the  usefulness  of  the  model  in  guiding  information  selection  in 
other  continuous  decision  and  control  situations.  The  work  reported 
here  builds  on  these  findings  by  experimental ly  investigating  the  effec¬ 
tiveness  of  the  model  for  real-time  information  selection  and  presenta¬ 
tion  and  for  evaluating  information  display  configurations. 

1.2  The  Problem 


Future  aircraft  will  be  charcterized  by  high  information  loads,  severe 
time  contraints,  and  complex  decisions  regarding  allocation  of  the 
operator's  attention  as  well  as  display  resources.  The  problems  ad¬ 
dressed  by  this  research  stem  from  these  increasing  supervisory  loads 
imposed  on  the  pilot  in  advanced  aircraft  operations.  This  is  a  prime 
example  of  the  larger  problem  facing  virtually  all  modern  military  com¬ 
mand  and  control  system--processing  and  selection  among  the  ever  in- 
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creasing  amounts  of  information.  Local  and  remote  computerized  systems 
make  available  copious  amounts  of  information  concerning  remaining 
resources,  environmental  state,  potential  computer  aiding,  and  predicted 
circumstances  and  actions.  In  such  cases,  the  costs  of  communications 
and  the  limited  processing  capabilities  of  the  human  operator  make  it 
necessary  to  optimize  the  information  selected,  processed  and  displayed. 

The  central  problem  in  performing  an  analysis  of  information  needs  is 
the  structuring  of  the  decision  process.  On  the  one  hand,  choices  need 
to  be  modeled  regarding  continuous  variables  such  as  the  flow  of  infor¬ 
mation:  sensing,  processing,  encoding,  transmitting  and  display  at  any 
point  in  time.  On  the  other  hand,  event-driven  parallel  activities  from 
situation  assessment  to  execution  strategies  must  be  described,  along 
with  such  situational  criteria  as  system  confidence,  operator  load  and 
capability,  and  task  characteristics.  This  R&D  program  represents  an 
effort  to  develop,  integrate  and  implement  models  and  techniques  that 
represent  and  enhance  the  airborne  information  handling  tasks. 

1.3  Technical  Approach 

In  brief,  the  information  handling  tasks  in  command  and  control  situa¬ 
tions  can  be  represented  as  a  multi-level,  multi-stage  decision  task  of 
information  acquisition  and  action  selection  (Chu,  Steeb  and  Freedy, 
1980).  At  each  stage  of  each  level,  the  decision  of  what  information  to 
display  reflect  the  task  circumstances,  the  operator's  and  automatic 
system's  capabilities,  and  the  communication  channel  characteristics. 
These  decisions  can  be  expressed  analytically  using  three  different 
methodologies:  (1)  production  rules  guided  by  pattern-directed  process 
control,  (2)  multi -dimensional  sets  of  utilities  tied  to  the  potential 
action  consequences,  and  (3)  sets  of  weighted  criterion  functions 
represented  in  terms  of  state-space  variables. 


The  previous  MAU-based  information  model  (Steeb,  Chu,  Clark,  Alperovitch 
and  Freedy,  1979)  handled  the  single-stage  decisions  present  in  airborne 
operations,  but  did  not  deal  with  the  many  continuous  behaviors  present 
in  monitoring  and  sampling.  Many  of  these  continuous,  stochastic 
processes  can  be  modeled  by  embedding  the  MAU  decision  model  in  a  queue¬ 
ing  model.  Here  the  time  distributions  of  event  arrival  and  the  demand¬ 
ed  attention  for  information  handling  can  be  postulated  and  measured  and 
queues  of  potential  messages  or  sampling  options  are  presented.  The 
multi -attribute  decision  model  is  then  incorporated  as  a  criterion  func¬ 
tion  in  the  queueing  model. 

The  model  evaluation  was  arcomplished  by  determining  the  effectiveness 
and  capabilities  provided  in  an  advanced  aircraft  task  situation.  This 
experimental  situation  included  both  multiple  threat  intercept  and  mul¬ 
tiple  subsystem  monitoring  tasks  in  a  continuous-time,  multiple-display 
simulation.  Also  provided  in  the  experimental  situation  were  extensive 
data  collection  and  performance  analysis  capabilities.  An  experiment 
was  conducted  based  on  this  multi-task,  airborne  information-handling 
situation  with  different  aiding  and  demand  levels.  Model -based  data 
aggregation  and  option  recommendations  led  toward  superior  performance, 
as  well  as  favorable  subjective  ratings.  The  model  also  provided  an 
analytical  framework  that  was  helpful  in  identifying  the  individual 
operator's  information  strategies. 

1.4  Current  Objectives 


The  focus  of  the  work  reported  here  is  the  demonstration  and  study  of 
the  MAU/queueing  model  as  an  expanded  version  of  an  adaptive  decision 
model  in  a  real-time  monitoring  and  control  environment.  The  specific 
objectives  that  were  addressed  include: 
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(1)  Develop  a  real-time,  multi-task  experimental  situation 
representative  of  an  information  handling  environment  in 
advanced  aircraft. 

(2)  Implement  the  MAU/queueing  model  in  an  experimental  situa¬ 
tion.  Perform  full  scale  experimental  study  of  model  ef¬ 
fectiveness. 

(3)  Determine  experimentally  the  effect  of  model  confidence, 
aiding  form,  and  task  demands  on  operator  acceptance  of 
aiding . 

(4)  Identify  forms  of  model-based  aiding  obtainable  using  the 
MAU/queueing  formulation.  Develop  a  set  of  transition 
rules  for  controlling  the  form  and  level  of  aiding  in 
response  to  operator  needs,  tasks  demands,  and  model  confi 
dence. 

1.5  Applications 


The  combined  approach  of  adaptive  information  value  estimation  and 
real-time  estimation  of  information  traffic  appears  to  be  most  applica¬ 
ble  to  decision  tasks  featuring  some  or  all  of  the  following  operational 
characteri sties : 

(1)  High  Information  Load.  The  operator  is  in  a  time-stressed 
decision  task.  For  each  decision,  he  can  process  only  a 
portion  of  the  available  data  set  and  must  choose  an  action 
within  a  short  time. 
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(2)  Costly  Information  Transmission.  The  transmission  of  data 
to  the  operator  is  subject  to  cost,  risk  of  detection,  or 
limited  transmission  capabilities.  Immediately  valuable 
information  must  be  selected. 

(3)  Significant  Judgmental  Factors.  The  decision  maker  must 
consider  the  credibility  and  content  of  the  evidence  along 
with  the  probabilities  and  utilities  of  the  consequences 
associated  with  each  ensuing  action. 

(4)  Multiple  Competing  Information  Sources.  A  variety  of  di- 
ferent  systems  or  sensors  must  be  monitored,  and  each  unat¬ 
tended  system  increases  in  uncertainty  over  time. 

Among  the  examples  of  actual  military  decision  making  situations  that 
require  such  tasks  are: 

(1)  Supervisory  control  and  decision  making  in  advanced  air¬ 
craft. 

(2)  Air  traffic  control. 

(3)  Remotely  piloted  fleet  guidance  and  control. 

(4)  Supervision  of  distributed  subsystems  and  platforms. 

(5)  Satellite  intelligence  coordination. 

(6)  Supervision  of  air,  ground,  or  sea  support  operations. 
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2.  MODELING  AND  SIMULATION  OF  INFORMATION  HANDLING  TASKS 


2.1  Overview 


The  activities  surrounding  the  selection  and  pi ^cessing  (acquisition  and 
dispatching)  of  sensed  information  often  constitute  a  majority  of  the 
operator's  supervisory  functions  in  advanced  aircraft.  The  operator 
must  maintain  an  awareness  of  the  sensed  environment,  the  machine 
states,  the  capacity  and  quality  of  the  communication,  and  the  progress 
toward  the  overall  objectives.  This  chapter  will  explore  the  techniques 
for  the  modeling  and  simulation  of  these  information  related  activities, 
which  is  subsumed  under  the  term  "information  handling"  in  this  study. 

In  particular,  the  selection  and  presentation  of  information  in  tactical 
airborne  systems  provides  a  prime  example  of  a  new  information  manage¬ 
ment  demand.  The  analysis  of  information  handling  in  supervisory  air¬ 
borne  systems  will  be  discussed  in  Section  2.2.  Further,  activities  in 
information  handling,  selection,  acquiring  and  processing,  can  be 
described  only  as  multiple  concurrent  proceses,  resulting  from  interac¬ 
tion  with  multiple  targets,  multiple  sensor  sources,  multiple  subsystems 
and  multiple  stages.  Models  of  the  operator's  information  handling  task 
performance  would  be  of  use  both  in  describing  the  interaction  between 
operator  and  system  and  in  predicting  the  performance  gains  to  be  ex¬ 
pected  from  the  introduction  of  varying  levels  of  computer  aiding. 
Further,  in  situations  in  which  the  responsibilities  for  some  tasks  are 
shared  by  other  crew  members  or  by  an  automated  decision  maker,  these 
models  might  also  be  used  within  the  system  to  coordinate  the  actions 
among  the  decision  makers.  The  modeling  approach  of  information  han¬ 
dling  tasks  will  be  described  in  Section  2.3. 
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The  simulation  approach  presented  in  this  chapter  expands  the  previous 
time-variant,  event-paced  information  value  model  into  the  operational 
domains  of  time-varing  information  characteristics  and  sporadic  event 
occurrence.  Here  the  time  distributions  of  physical  events,  such  as 
system  faults,  course  errors,  and  threat  arrival,  can  be  estimated.  The 
process  model  embedded  in  the  queueing  framework  provides  all  the  neces¬ 
sary  updates  of  subjective  and  objective  information  value  estimates. 

The  combined  MAU/queueing  model  simulation  is  used  in  depicting  the 
operator's  continuous  monitoring  and  control  functions  as  standard  in¬ 
formation  handling  tasks  ranging  from  situation  assessment  to  intermit¬ 
tent  control.  The  simulation  of  an  information  handling  task  will  be 
outlined  in  this  chapter  and  further  described  in  Chapter  3. 


Supervisory  I n f o rmation  Handlinc 


The  initial  idea  of  supervisory  control  narrowly  referred  to  the  task  of 
monitoring  automatically  controlled  processes  and,  when  necessary,  in¬ 
tervening  and  adjusting  reference  points.  As  an  example,  piloting  an 
aircraft  requires  monitoring  the  aerodynamic  configuration  to  ensure 
that  the  autopilot  is  working,  trimming  the  set  points  to  compensate  for 
disturbance,  and  intervening  in  the  case  of  autopilot  failures  and  emer¬ 
gencies.  When  a  process  is  automated  or  semi -automated,  the  control  ac¬ 
tions  need  not  be  continuously  produced,  and  the  opertor  need  not  devote 
full  attention  to  that  process.  This  makes  it  possible  for  the  operator 
to  be  responsible  for  multiple  proceses.  As  a  result,  the  narrow  sense 
of  supervisory  control  can  be  expanded  to  include  a  broad  spectrum  of 
concurrent  activities  from  control  to  supervision  and  planning.  This 
broad  sense  of  supervisory  control  can  be  defined  as  "Managing  and  con¬ 
trolling  a  semi -autonomous  system  through  the  intermediary  of  a  comput¬ 
er.  The  human  supervisor  performs  upper-level  goal -oriented  functions 
such  as  planning  system  activities,  programming  the  computer,  monitoring 
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the  system  behavior  when  computer-controlled,  adjusting  parameters  on¬ 
line  when  appropriate,  and  intervening  to  take  over  control  in  an  emer¬ 
gency  or  for  normal  reprogramming  or  repair."  (Seifert,  1979). 

2.2.1  Supervisory  Handling  Paradigm.  In  the  context  of  supervisory 
control  functions  in  advanced  aircraft,  the  pilot  or  the  operator  as  the 
airborne  system  manager  faces  a  variety  of  information  sources  and 
displays--such  as  a  master  monitor  display,  and  an  integrated  multifunc¬ 
tion  display.  These  displays  may  be  event  driven  or  functional  or  pro¬ 
cedural.  The  pilot  has  the  responsibility  to  (1)  monitor  the  aircraft 
subsystems,  detect  possible  hardware  failures  and  potential  hazards;  (2) 
respond  to  events  such  as  communication  of  tactical  information,  change 
of  aircraft  configuration,  and  reduction  of  4-D  guidance  errors;  and  (3) 
react  to  unexpected  events  such  as  identification  of  threat,  change  of 
flight  plan,  establ i shment  of  the  backup  mode,  and  declaration  of  emer¬ 
gencies,  etc. 

If  the  task  is  viewed  as  the  totality  of  the  situational  information  and 
required  actions  imposed  on  the  pilot,  it  appears  that  complete  super¬ 
visory  handling  task  descriptions  should  include  interactive  activities 
of  the  operator  in  defining,  initiating  and  monitoring  subtasks.  While 
the  pilots  usually  accept  the  overall  goal  and  plan  as  instructed,  the 
process  of  organizing  the  activities  of  situation  assessment  and  dynamic 
replanning  at  the  top  level,  status  monitoring  and  action  execution  at 
the  intermediate  level,  and  observation  and  control  at  the  bottom  level, 
is  a  continued  and  recurrent  one.  The  three-level  hierarchy  of  super¬ 
visory  information  handling  is  described  in  Figure  2-1. 

The  major  subprocesses  involved  in  situation  assessment  in  the  context 
of  supervisory  control  of  advanced  aircraft  include  the  following: 


2-3 


FIGURE  2-1. 

SPECTRUM  OF  OPERATOR  ACTIVITIES  IN 
SUPERVISORY  CONTROL  OF  ADVANCED  AIRCRAFT 
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(1)  Event  sensing:  directing  sensors,  establishing  communica¬ 
tion  and  managing  data  flows. 

(2)  Data  fusion:  generating  hypotheses  and  updating  situations 
estimates. 

(3)  Problem  recognition:  identifying  conflicts  or  problems  re¬ 
quiring  resolution. 

(4)  Requirement  analyses:  defining  actual  activities  required 
and  possible  intervening  functions. 

The  situation  assessment  activities  are  tightly  coupled  with  the  dynamic 
planning  activities.  These  planning  activities  include  procedure  visu¬ 
alization,  subtask  sequencing,  resource  distribution,  and  outcome  pred¬ 
iction.  Most  of  these  high-level  activities  are  realized,  as  shown  in 
Figure  2-1,  at  a  group  of  a  lower  level  of  activities  (the  monitoring 
action  pair).  The  major  monitoring  processes  include: 

(1)  Feature  extraction:  partitioning  information  into  subsets, 
assessing  the  relevance  of  a  particular  information  set. 

(2)  Model  matching:  refining  the  hypothesis  at  the  appropriate 
level  of  detail.  Weighting  the  importance  of  situation 
states. 

(3)  Signal  sampling:  updating  probability  estimates. 
Reassessing  the  values  and  utility  of  potential  outcome. 

The  action  process,  on  the  other  hand,  deals  with  response  generation 
and  execution  issues.  This  monitoring-action  loop  represents  the  major 
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portion  of  in-task  information  handling,  since  many  assessment-planning 
activities  are  performed  in  a  pre-task  fashion,  and  the  trend  toward  fu¬ 
ture  aircraft  suggests  that  pilots  will  be  less  and  less  concerned  with 
manual  observation  and  control. 

At  the  lowest  level  of  the  task  hierarchy  is  the  observation  and  control 
pair,  which  require  continuous  scanning,  acquiring,  and  tracking  of  a 
specific  target  or  event  of  importance.  Usually  this  lowest  level  in¬ 
volves  the  highest-frequency  functions,  such  as  target  track  and  vehicle 
stabilization.  These  functions  are  typically  automated,  but  unexpected 
events  and  system  malfunctions  may  arise,  making  the  operator  back-up 
activities  crucial.  In  the  following  discussion,  we  shall  concentrate 
the  intermediate  level  of  the  supervisory  control  paradigm. 

2.2.2  Supervisory  Information  Characteristics.  In  general,  the  in¬ 
teractive  supervisory  information  handling  involves  one  or  many  of  the 
following  activities: 

(1)  Continually  sense  and  symbolically  characterize  all  infor¬ 
mation  with  respect  to  the  operation  ranges  and  their  rela¬ 
tive  importance. 

(2)  Continually  update  situational  hypotheses,  by  hypothesis 
formation,  model  parameter  selection  and  experimentation 
(accessing  information). 

(3)  Systematical ly  relate  maneuvers  and  action  sequences  to  ve¬ 
rifiable  consequences. 

(4)  Provide  consistent  interpretation,  by  keeping  track  of 
focus-of-attention  (e.g.,  target)  and  resolving  ambiguities 
or  contradictions. 
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In  our  previous  study  (Chu  et  al - ,  1980),  the  handling  features  were 
analyzed  along  decision  dimensions,  characterizing  the  situation  by,  for 
example,  the  danger  or  frequency  of  threats,  the  time  available  for  de¬ 
cision  making,  and  the  options  and  characteristics  of  information  con¬ 
cerning  the  aircraft  and  the  environment. 

Timel ines.  The  information  available  at  a  given  time  is  dependent  on 
the  environmental  situation,  the  sensor  characteristics,  the  data  base 
content,  and  the  display  capabilities.  The  information  itself  may  con¬ 
sist  of  data  regarding  weather  conditions,  aerodynamic  status,  target 
track,  ECM,  and  mission  status. 

Handling  Cost.  The  costs  of  acquiring  information  result  from  the  sen¬ 
sor  characteristics,  the  direct  and  indirect  costs  of  sensor  deployment, 
information  processing  and  display,  and  the  amount  of  attention  the 
operator  can  contribute.  The  direct  costs  of  information  acquisition 
include  such  factors  as  energy  expenditures  and  equipment  expenses.  In¬ 
direct  costs  include  increased  possibilities  of  detection  and  counter¬ 
measures.  The  available  operator  attention,  finally,  is  defined  by  the 
task  demands  and  the  individual  capabilities  of  the  operators. 

Expected  Payoffs.  The  costs  and  payoffs  associated  with  the  myriad  pos¬ 
sible  outcomes  vary  with  mission  phase.  The  consequences  are  defined 
not  only  in  terms  of  equipment  and  attainment  of  objectives,  but  also  as 
a  function  of  organizatiuonal  policy  and  procedures.  The  relative  im¬ 
portance  of  fuel  expenditures,  vehicle  survival  countermeasures,  etc., 
change  as  the  mission  objective  is  approached,  attained,  or  past.  The 
relative  importance  of  these  factors  must  be  assigned  by  the  human 
operator  or  by  the  command  group. 
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Time  Stress.  Available  time  for  decision  making  varies  throughout  the 
mission  as  a  direct  function  of  the  varying  vehicle  speed,  altitude,  and 
surrounding  weather  conditions.  Altitude,  cloud  cover  and  ECM  determine 
the  distance  from  which  obstacles,  navigation  points,  or  targets  can  be 
observed.  The  speed  then  determines  the  available  time.  Decision  time 
can  be  expected  to  influence  the  amount  of  information  that  can  be  pro¬ 
cessed  and  the  probability  distribution  of  the  possible  consequences. 

The  above  analysis  is  based  on  a  multiple-threat  intercept  scenario 
developed  in  a  previous  study.  A  separate  analysis  was  performed  based 
on  a  flight  management  scenario  by  Rouse  and  Neubauer  (1978)  and  result¬ 
ed  in  a  similar  set  of  information  requirements:  (1)  type  of  informa¬ 
tion  (reports  vs.  forecasts),  (2)  time  period,  (3)  uncertainty,  (4) 
cost  (fixed  vs.  recurring),  and  (5)  format.  As  the  purpose  of  their 
study  was  to  define  the  attributes  of  an  information  system  from  the 
design  point-of-view,  the  following  interpretation  of  the  above  informa¬ 
tion  characteristics  were  given.  Type  of  information  includes  reports 
of  what  has  happened  or  is  happening,  and  forecasts  of  what  might  hap¬ 
pen.  Time  period  refers  to  the  time  interval  that  the  report  or  fore¬ 
cast  covers.  Uncertainty  reflects  the  finite  sample  sizes  upon  which 
reports  are  based  or  the  likelihood  that  forecasts  are  based  on  imper¬ 
fect  models.  Cost  can  vary  in  type  and  magnitude.  Fixed  costs  apply  to 
information  sources  that  incur  no  additional  cost  after  they  have  been 
initially  acquired.  Recurring  costs  are  those  that  are  incurred  every 
time  an  information  source  is  used.  Some  information  sources  may  in¬ 
volve  both  fixed  and  recurring  costs.  Reduction  in  uncertainty  may  in¬ 
cur  extra  costs.  Thus  information  cost  and  uncertainty  involve  a  tra¬ 
deoff  for  a  manager.  The  format  of  an  information  system  involves  the 
way  in  which  the  manager  queries  the  system  and  the  way  in  which  the 
system  responds.  Examples  cited  include  menu  keyboard,  voice,  or  free 
format  such  as  graphics  and  other  physiological  links. 
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2.2.3  Categories  of  Aiding  in  Information  Handling.  The  various  han¬ 
dling  subprocesses  described  in  the  previous  sections  indicate  the  pos¬ 
sibility  of  aiding  the  operator  at  a  number  of  levels.  The  levels  range 
from  simple  aggregation  to  complete  information  system  management,  and 
different  ones  may  be  invoked  according  to  the  situational  demands  and 
operator  needs.  The  levels  correspond  roughly  to  the  five  levels  of  au¬ 
tomation  recommended  for  future  airborne  information  management  systems 
(Mertes  and  Jenney,  1974).  The  various  levels  are  described  below  in 
order  of  increasing  complexity. 

Aggregation.  The  decision  aiding  system  has  access  to  event  likeli¬ 
hoods,  situational  data,  and  preference  data.  Aggregated  information 
may  be  abstracted  and  presented  to  the  operator  for  use  in  decision  mak¬ 
ing.  For  example,  the  system  may  ascertain  the  immediate  likelihood  of 
enemy  threats,  the  expected  effectiveness  of  an  avoidance  maneuver,  or 
the  fuel  consumption  anticipated  for  a  climbing  attack.  A  number  of 
probability  aggregation  displays  have  been  demonstrated  in  making  diag¬ 
nostic  decisions  about  reconnaissance  data  (Howell,  1967).  Howell 
states  that  improvements  in  diagnostic  decisions  of  about  10-15%  can  be 
expected  with  automated  aggregation.  Improvements  become  particularly 
noticeable  under  conditions  of  time  or  load  stress  or  low  input  fidelity 
(Kelly  and  Peterson,  1971). 

Alerting.  The  aiding  system  may  sense  an  out  of  threshold  condition  re¬ 
quiring  operator  intervention.  An  alerting  display  can  be  shown  to  the 
operator  along  with  a  description  of  the  problem.  This  is  especially 
important  for  monitoring  of  infrequent  or  long  duration  events  (Mertes 
and  Jenney,  1974).  Again,  no  action  recommendation  is  made,  although 
explanations  may  be  provided. 
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Option  Recommendation.  The  aiding  system  may  recommend  information  to 
acquire  or  actions  to  execute.  The  multi-attribute  model  represents  the 
policy  of  the  specific  user.  It  has  access  to  the  factors  characteriz¬ 
ing  each  information  choice,  and  it  has  inputs  from  the  queueing  model. 
The  model  can  thus  be  configured  to  scan  the  available  information 
sources  and  action  options,  and  recommend  the  immediately  most  effective 
choice. 

Because  of  the  "look-ahead"  approach  implicit  in  the  aiding  system's  de¬ 
cision  tree,  the  reasons  for  alternative  selection  can  be  automatically 
generated  and  presented  to  the  operator.  For  example,  in  a  multiple  in¬ 
tercept  situation,  the  pilot  may  want  to  know  why  the  computer  recom¬ 
mended  a  change  to  low  azimuth  track-while-scan  radar  mode.  Rather  than 
simply  giving  numerical  comparisons,  which  are  hard  to  decipher,  the  de¬ 
cision  tree  may  be  analyzed  and  translated  into  meaningful  information 
about  likelihoods  of  target  acquisitions  and  the  effects  of  associated 
engagement  maneuvers. 

Information  Management.  The  functions  of  recommendation  may  be  extended 
to  automation  by  linking  the  aiding  system  to  the  onboard  information 
control  system.  The  model  may  be  used  to  direct  the  sensing  systems,  to 
select  information,  and  format  the  data  display  to  the  opertor.  This 
process  may  be  accomplished  through  use  of  weighting  matrices  specifying 
the  effects  of  the  various  information  sensing  and  formatting  choices  on 
the  fol lowing : 

(1)  Event  Sensing.  This  aspect  of  the  information  handing  pro¬ 
cess  involves  the  sensing  and  communication  of  environmen¬ 
tal  conditions,  threat  type  and  location,  own  force  status 
and  other  relevant  information.  The  sensors  may  include 
video,  infra-red,  radar  and  on  board  detection  systems. 
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The  information  may  need  to  be  routed  adaptively  from  the 
point  of  acquisition  to  the  appropriate  processing  and 
display  unit. 

(2)  Data  Fusion.  Data  fusion  is  the  aggregation  of  the  infor¬ 
mation  obtained  from  the  various  sensors  and  the  generation 
and  testing  of  situational  hypotheses.  In  this  way,  the 
local  or  global  situation  estimate  is  updated  using  produc¬ 
tion  rules  or  Bayesian  methods. 

(3)  Problem  Recognition.  Problem  recognition  involves  the  mon¬ 
itoring  of  tolerance  ranges  around  critical  variables  (fuel 
limitation,  flight  surface  constraints,  launch  acquisition 
region,  etc.)  to  determine  if  correct  actions  need  to  be 
initiated.  Problem  recognition  in  aircraft  supervision  in¬ 
volves  tolerance  checks  either  on  the  current  state  or  on 
the  predicted  future  state.  The  aiding  system  might,  for 
example,  use  models  of  the  operator  to  detect  and  warn  of 
inconsistencies  in  the  operator's  performance. 

(4)  Source  Selection.  Simultaneous  consideration  of  multiple 
alternatives  portrayed  against  multiple  criteria  quickly 
becomes  too  complex  for  the  opertor  to  resolve.  Computer 
based  aggregation  of  the  various  factors  is  typically  fas¬ 
ter  and  more  consistent  than  is  possible  by  the  operator. 

The  multi -attribute  utility  model  represents  the  policy  of 
the  specific  user;  it  has  access  to  the  factors  character¬ 
izing  each  information  choice,  and  it  can  be  linked  to  the 
on-board  information  control  system.  The  model  can  be  con¬ 
figured  to  automatically  scan  the  available  information 
source,  select  the  immediately  most  useful  source,  and 

and  display  it  to  the  operator. 
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(5)  Event  Sequencing.  Once  data  has  been  collected  from  the 
sensors,  situation  estimates  updated,  and  potential  prob¬ 
lems  recognized,  then  procedures  associated  with  the  event 
must  be  sequenced  that  will  resolve  conflicts  and  achieve 
goals.  The  sequenced  actions  may  be  synthesized  using  one 
or  more  of  the  following  approaches:  (a)  means-ends 
analysis,  (b)  backtracking,  (c)  hierarchical  planning,  and 
(d)  production  system;  or  using  simply  the  priority  rules 
or  queing  di scipl ines. 

(6)  Consequence  Evaluation.  Implicit  in  the  value-based  event 
sequencing  is  the  concept  of  an  evaluation  function-- 
maximization  of  operation  time,  optimization  of 
jammimg/communication,  etc.  Each  candidate  course  of  ac¬ 
tion  should  be  scaled  along  the  common  set  of  criterion  di¬ 
mensions.  Weighting  of  the  choices  in  importance  then  al¬ 
lows  systematic 'comparison  of  the  possible  action  choice. 

If  the  criterion  dimensions  are  probabilistic  in  nature, 
e.g.,  detection  of  communications,  tactical  gains,  or 
losses  sustained,  then  the  expectation  of  the  outcome  is 
used  in  the  evaluation.  Additional  risk  factors  can  be  ad¬ 
ded  if  the  selection  policy  is  not  risk  neutral . 

(7)  Event  Selection.  Event  (procedure)  selection  involves  the 
comparison  by  the  affected  subprocesses  of  all  candidate 
events,  since  the  affected  subprocesses  may  have  different 
goal  sets,  and  may  compete  for  resources  under  different 
criterion  functions.  In  this  case,  it  is  necessary  to  bar¬ 
gain  or  to  select  according  to  aggregated  value  judgment. 
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Information  Handling  Task  Model  for  Supervisory  Control 


2.3.1  General .  Analysis  of  the  supervisory  information  handling 
tasks  and  the  aiding  categories  described  in  the  previous  section  has 
supported  the  needs  for  modeling  the  recurrent  decisions  of  selecting 
information  sources.  A  particularly  attractive  approach  is  one  that  in¬ 
corporates  the  key  factors--ai rcraft  state,  environmental  conditions, 
operator  capabilities,  acquisition  costs,  etc. --into  a  mul ti -attribute 
deci sion  model .  This  individualized  model  of  information  seeking  policy 
has  been  found  to  be  useful  for  evaluating  alternative  information  sys¬ 
tem  configurations  and  for  automating  the  information  handling  task. 

The  models  have,  for  simplicity,  been  time-invariant  and  driven  by  paced 
events.  The  decisions  have  been  indiscriminantly  triggered  by  sensing 
an  environmental  obstacle  or  threat.  The  model  then  selects  the  most 
effective  information  source  for  dealing  with  the  unequivocal  event. 

Information  handling  in  advanced  aircraft  also  concerns  process  aspects 
such  as  the  sequencing  of  information  handling,  especially  when  there 
are  a  number  of  continuous  processes--supervi sion  of  subsystems,  commun¬ 
ications,  aerodynamic  surfaces  and  multiple  threats,  etc.  Many  of  these 
time  continuous  processes  can  be  considered  as  separate  classes  of  event 
queues  for  pilot  decisions.  Even  the  most  complicated  procedures  in  in¬ 
formation  handling  can  then  be  modeled  as  a  network  of  sequences  that  as 
a  whole  guides  the  information  flow  from  system  to  operator.  The  ap¬ 
proach  adopted  in  this  study  expands  the  previous  time  invariant, 
event-paced  information  model  into  the  operational  domains  of  time  vary¬ 
ing  information  characteristics  and  sporadic  event  occurrence.  The  time 
distributions  of  aircraft  environment  and  physical  events,  such  as  sys¬ 
tem  faults,  course  errors,  and  threat  assessment  are  estimated,  and  up¬ 
dates  of  subjective  information  values  and  objective  information  attri¬ 
butes  are  evaluated.  Central  to  this  evaluation  study  is  the  marriage 
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of  the  evaluation  model  and  the  process  model  and  a  method  based  on  the 
use  of  multi -attribute  utility  theory  and  queueing  theory  to  organize 
the  various  objective  and  subjective  factors  that  enter  into  the  infor¬ 
mation  handling  decision.  The  concept  for  the  modeling  of  supervisory 
information  handling  is  summarized  in  Figure  2-2.  The  multi -attribute 
utility  model  and  the  continuous  information  process  model  each  will  be 
discussed  in  the  following  sections. 

2.3.2  Multi -Attribute  Utility  Model.  Use  of  the  mul ti -attribute 
utility  (MAU)  models,  pioneered  by  Raiffa  and  his  colleagues  (Raiffa, 
1969;  Keeney  and  Raiffa,  1975)  and  by  V.  Winterfeldt  (1975)  makes  the 
information  handling  process  more  goal  directed,  normative  and  axiomat¬ 
ic.  Instead  of  simply  attempting  to  prer  :ct  behavior  on  the  basis  of  a 
set  of  independent  features,  the  utility  models  tie  the  information  de¬ 
cisions  directly  to  the  ensuing  action  decisions.  The  value  of  obtain¬ 
ing  information  is  determined  by  calculating  its  impact  on  the  expected 
utility  of  the  subsequent  action  decision.  The  information  is  assumed 
to  change  the  probability  distributions  of  the  consequence  set  and,  in 
turn,  to  revise  the  expected  values  of  the  alternative  actions. 
Nevertheless,  the  form  of  the  model  is  simply  a  linear  additive  rule. 

The  utility  of  an  action  is  considered  to  be  an  aggregate  of  many  possi¬ 
ble  outcomes,  each  expressed  along  a  set  of  attributes: 


EU  (ak)  =  t  P(zt)  l  U1(ak.  zj 

states  attributes 

a  i 


ity  of  state  z%  occurring,  and  U^a^,  z^)  is  the  utility  function  over 
the  i ^  attribute  associated  with  state  l  and  action  k.  The  formulation 
is  the  result  of  several  key  simplifying  assumptions.  The  decision  mak¬ 
er  is  assumed  to  be  risk  neutral,  so  that  he  is  indifferent  between  the 
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expectation  across  a  set  of  uncertain  outcomes  and  the  uncertain  out¬ 
comes  themselves.  This  allows  the  probabilities  to  be  entered  as  simple 
coefficients.  Also,  the  attributes  are  assumed  to  satisfy  additive  in¬ 
dependence,  allowing  the  linear  additive  form  of  aggregation.  Tests  for 
compliance  with  these  assumptions  can  be  found  in  V.  Winterfeldt  (1975) 
or  Keeney  and  Raiffa  (1976). 

The  impact  of  a  message  or  item  of  data  is  to  change  the  probability 
distribution  of  the  states  zv  Once  the  message  is  received,  a  maximum 
utility  action  a*(y)  can  be  identified.  The  expected  utility  of  select¬ 
ing  an  information  source  S  then  becomes  (Emery,  1969): 

EU  (S)  =  l  l  P(z£)  P(y[zA)  u(a*(y),zi) 

messages  states 
y  z 

Here,  u(a*(y),z^)  is  the  utility  of  taking  action  a*(y)  given  that  state 
z^  occurs.  The  utility  function  is  again  mul ti -attributed ,  but  for  sim¬ 
plicity,  u(a*(y),zi)  is  portrayed  as  having  already  been  aggregated 
across  the  various  dimensions. 

This  type  of  analysis  is  suited  for  highly  structured  tasks.  Not  only 
must  the  possible  states,  messages,  actions,  and  outcomes  be  specifi¬ 
able,  but  the  prior  state  probabilities  and  the  conditional  probabili¬ 
ties  characterizing  the  information  system  must  be  derivable.  The  se¬ 
quence  of  decision  stages  can  be  depicted  using  a  decision  tree,  as 
shown  in  Figure  2-3.  The  tree  is  folded  back  by  associating  with  each 
possible  message  the  maximum  expected  utility  of  the  subsequent  actions. 
This  folding  back  represents  graphically  the  process  of  EU  maximization. 
The  favored  information  source  S  is  then  identified  by  comparing  the  ex¬ 
pectations  taken  over  all  possible  messages. 


2-16 


FIGURE  2-3. 

DECISION  TREE  FOR  INFORMATION  SEEKING 


Factor  Choice.  As  described  above,  the  multi-attribute  decision  model 
of  information  seeking  behavior  is  based  on  a  weighted  aggregation  of 
the  factors  which  enter  into  a  decision.  In  its  simplest  form  the 
aggregation  is  a  linear  additive  model: 

HAU  (ak)  .  I  «,  1),  (X1k) 

attributes 

i 

where  =  £  P(zJi)X1-j£ 

states 

i 

where  MAU  (a|J  is  the  aggregate  utility  of  action  k 
Wj  is  the  importance  weight  of  attribute  i 
X-j|<  is  the  level  of  attribute  i  associated  with  action  k 
P(za  )  is  the  probability  of  occurrence  of  state  zz 

The  choice  of  factors  or  attributes  in  this  model  is  extremely  impor¬ 
tant.  It  was  noted  in  our  earlier  study  (Steeb,  Chen  and  Freedy,  1977) 
that  the  attribute  set  should  be  accessible,  monotonic,  independent, 
complete  and  meaningful.  Also,  a  single  set  must  account  for  both  in¬ 
formation  acquisition  and  action  selection  behavior.  Finally,  the  at¬ 
tribute  set  must  be  manageably  small  in  dimension.  With  these  con¬ 
siderations  in  mind,  an  initial  taxonomy  of  consequences  can  be  organ¬ 
ized  around  the  following  five  areas: 

(1)  Communication  costs--such  as  energy,  equipment,  and  atten¬ 
tion. 
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(2)  Equipment  attrition--fuel  expenditures,  probability  of 
vehicle  damage,  etc. 

(3)  Objectives  attai nment--area  reconnoitered ,  payload 
del i vered . 

(4)  Dynamic  effects--effects  on  time  delay,  availability  of  fu¬ 
ture  information  and  system  capabilities. 

(5)  Subjective  factors--preferences  regarding  control  continui¬ 
ty,  operator  load. 

Attribute  Level  Determination.  The  level  or  quantity  of  each  attribute 
for  a  given  outcome  can  be  determined  in  several  ways.  For  example, 
mappings  between  predictive  features  and  the  attributes  can  be  esta¬ 
blished  by  observation  and  adjustment.  Here,  data  available  to  the  de¬ 
cision  program  concerning  the  environmental  state,  vehicle  state,  chan¬ 
nel  characteristics,  sensor  capabilities,  and  operator  load  can  be  used 
td  predict  the  attribute  levels.  Alternatively,  the  attribute  levels 
may  be  estimated  subjectively  or  established  from  performance  histories. 
Use  of  mappings  from  predictive  features  is  more  attractive  than  subjec¬ 
tive  estimates  as  no  load  is  imposed  on  the  operator,  and  situation- 
specific  factors  may  be  taken  into  account.  For  example,  the  communica¬ 
tion  delay  may  be  directly  predicted  from  sensor  queue  length,  sensor 
response  characteristics,  and  transmission  distance.  Subjective  esti¬ 
mates  or  pre-established  values  for  the  attribute  levels  would  tend  to 
be  much  less  reliable  than  such  in-task  calculations. 

Attribute  Weight  Estimation.  The  operator's  goal  structure  and  policy 
for  information  handling,  expressed  as  importance  weights,  could  be  eli¬ 
cited  or  inferred  and  then  incorporated  in  the  model.  There  are  a 
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number  of  advantages  to  such  subjective  estimation,  particularly  with 
respect  to  allocation  of  function.  By  incorporating  individualized 
operator  weights  in  the  model,  the  complex  evaluation  and  goal  direction 
functions  remain  the  responsibility  of  the  operator,  while  the  normative 
aggregation  functions  are  assumed  by  the  computer.  Also,  operator  ac¬ 
ceptance  of  aiding  by  the  model  may  be  increased  since  individual 
preferences  are  incorporated  in  the  machine  decisions. 

The  operator's  subjective  weights  may  be  defined  off-line  by  elicitation 
or  on-line  through  inference.  The  off-line  methods  include  direct  eli¬ 
citation  of  preference,  decomposition  of  complex  gambles  into  hypotheti¬ 
cal  lotteries,  and  use  of  mul ti -variate  methods  to  analyze  binary 
preference  expressions.  These  techniques  are  accurate  and  reliable  in 
many  circumstances,  but  they  have  a  number  of  disadvantages  when  applied 
to  operational  systems  (Chu  et  al ,  1980). 

Estimation  techniques  relying  on  inference  from  in-task  behavior  may  be 
more  useful.  The  inference  techniques  can  be  based  on  non-parametric 
f^ans  of  pattern  recognition.  Here  a  model  of  decision  behavior  is  as- 
s jmed  and  the  parameters  of  the  model  are  then  fitted  by  observation  and 
adjustment.  Briefly,  the  technique  developed  considers  the  decision 
maker  to  respond  to  the  characteristics  of  the  various  alternatives  as 
patterns,  classifying  them  according  to  preference.  A  linear  discrim¬ 
inant  function  is  used  to  predict  the  decision  maker's  choices,  and  when 
amiss,  is  adjusted  using  error  correcting  procedures. 

The  adaptive  nature  of  the  estimation  program  is  shown  in  Figure  2-4. 
Expected  consequence  vectors  associated  with  each  information  source  are 
input  to  the  model.  These  consequence  vectors  are  multiplied  with  the 
weight  vector,  resulting  in  evaluations  along  a  singlt.  utility  scale. 

The  maximum  utility  choice  is  determined  and  compared  with  the 


2-20 


INFO.  SOURCE  I— ~C 
INFO.  SOURCE  2— — C 

INFO.  SOURCE  n— •*[ 


FIGURE  2-4. 

ADAPTIVE  ESTIMATION 


EXPECTED 

CONSEQUENCE 

VECTORS 


ATTRIBUTE 

WEIGHT  EVALUATION 

VECTOR 


X,  X,  .  .  .  X,  ] 

x2  x'2  ...  Xj1 


|  ADJUSTMENT 
IK'  ■  K  +  Ux.-xJ 


OPERATOR 

CHOICE 


COMPARE 


MAUm 

tit 


/ 


MODEL  CHOICE 


MAX 


PROCESS 


\  j 

i. 


2-21 


operator's  actual  choice.  If  a  discrepancy  occurs,  the  weight  vector  is 
adjusted  according  to  the  following  rule: 

w'  =  w  + X  (xc  -  xm) 

where  w'  is  the  updated  weight  vector 
w  is  the  previous  weight  vector 
X  is  an  adjustment  constant 

xc  is  the  attribute  vector  of  the  chosen  alternative 

xm  is  the  mean  attribute  vector  of  all  alternatives  ranked 
by  the  model  above  the  chosen  alternative 


Ideally,  the  error  correction  moves  the  weight  vector  in  a  direction 
minimizing  subsequent  errors.  The  amount  of  movement  depends  on  x,  the 
adjustment  increment. 

The  utility  model  could  be  trained  subjectively  by  observation  of  the 
operator's  choices.  If  the  sequence  of  choices  led  to  an  objectively 
favorable  outcome,  the  trained  parameter  set  would  be  retained.  If  the 
outcome  was  unfavorable,  the  parameter  set  would  be  returned  to  the  lev¬ 
els  present  prior  to  the  sequence  of  decisions.  In  this  way,  objective 
criteria  would  guide  overall  training,  but  the  explicit  decision-by¬ 
decision  policy  for  information  management  would  be  subjectively 
derived . 

Experimental  Validation.  Evidence  for  the  usefulness  of  the  multi¬ 
attribute  utility  formulation  and  adaptive  estimation  programs  was  ini¬ 
tially  obtained  in  an  earlier  program  (Steeb,  Chen  and  Freedy,  1977).  A 
simulation  resembling  control  of  a  remotely  piloted  vehicle  (RPV)  was 
used  in  this  study.  The  adaptive  model  was  found  to  be  significantly 


more  predictive  of  the  subject's  behavior  than  either  a  constant,  unity 
weight  model  or  an  off-line  method  of  weight  estimation.  Also,  the 
model  was  found  to  be  useful  in  identifying  different  decision  policies 
or  styles. 

The  following  efforts  (Steeb,  Davis,  Alperovitch  and  Freedy,  1978; 

Steeb,  Chu,  Clark,  Alperovitch  and  Freedy,  1979)  re-directed  the  appli¬ 
cation  area  from  one  of  remotely  piloted  vehicle  supervision  to  one  of 
information  selection  in  advanced  aircraft.  A  simulation  based  on  mul¬ 
tiple  threat  intercept  operations  in  advanced  aircraft  was  developed. 
Comparisons  were  made  between  (1)  automated  information  selection  based 
on  the  adaptive  model  described  earlier,  (2)  automated  information 
selection  based  on  information  seeking  strategies  elicited  directly  from 
the  operator,  and  (3)  manual  information  selection.  The  adaptive  tech¬ 
nique  was  found  to  be  superior  to  direct  policy  elicitation,  both  for 
automated  information  selection  and  as  a  basis  for  information  system 
evaluation. 

2.3.3  Approaches  to  Model  the  Continuous  Information  Handling  Pro¬ 
cess.  There  are  two  general  approaches  to  model  continuous  information 
handling  behavior.  Descriptive  modeling  is  designed  to  describe  actual 
or  observed  behavior,  whereas  prescriptive  modeling  prescribes  optimal 
or  goal/criterion-directed  behavior.  As  rational,  effective  behavior  of 
the  operator  is  the  main  concern,  the  use  of  prescriptive  models  is  em¬ 
phasized  in  this  study. 

Prescriptive  models  are  often  based  on  optimization  of  underlying 
descriptive  models.  These  models  include  control  and  estimation  ap¬ 
proaches  (Govindaraj,  1979;  Rouse,  1980),  process  interaction  approaches 
(Seifert  and  Chubb,  1978;  Tulga  and  Sheridan,  1980),  and  heuristic-based 
approaches  (Wesson,  1977;  Goldstein  and  Grimson,  1977;  Engleman,  Berg, 
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and  Bischoff,  1979).  Based  on  the  context  of  task,  the  detail  level  in 
supervisory  control  and  the  characteri sties  of  the  task  demand,  it  ap¬ 
pears  that  optimal  controller  and  observer  models  are  suitable  for  the 
modeling  of  lowest-level,  supervisory  control  task  and  the  knowledge  and 
heuristic-based  techniques  can  be  successfully  applied  to  the  high- 
level,  situation  assessment  plan  generation  task.  At  the  intermediate 
level,  various  approaches  have  been  used  to  model  the  monitoring  and  ex¬ 
ecution  activities  of  information  handling.  These  approaches  will  be 
discussed  in  the  following  paragraphs. 

A  considerable  amount  of  research  effort  has  been  directed  at  under¬ 
standing  the  human  operator  in  multi-task,  supervisory  control  situa¬ 
tions.  One  general  approach  is  to  start  with  control  theory  from  a  gen¬ 
eral  perspective  that  includes  control  with  respect  to  continuous  events 
as  well  as  discrete  events.  This  approach  is  represented  by  Mu¬ 
ral  idharan  and  Baron  (1979),  Govindaraj  and  Rouse  (1979),  Krishna-Rao 
and  Ephrath  and  Kleinman  (1979). 

Mural idharan  and  Baron  (1979)  have  studied  supervisory  control  of  multi¬ 
ple,  remotely  piloted  vehicles  (RPVs),  The  operator  has  to  choose  which 
RPV  to  monitor  and  whether  or  not  to  intervene  with  discrete  corrective 
control  actions  based  on  the  venicle's  lateral  deviations.  Mural idharan 
and  Baron's  model  is  an  extension  of  the  optimal  control  model  of  the 
operator  derived  by  infusion  decision  theoretic  notions  into  control 
criteria.  The  model  has  been  employed  to  study  the  effects  of  error 
tolerance  and  the  number  of  RPVs  per  operator  on  overall  system  perfor¬ 
mance  in  terms  of  timing  errors  and  deviations  from  the  desired  trajec¬ 
tory. 

Govindaraj  and  Rouse  (1979)  have  studied  intermittent  control  with  a 
preview  of  map  displays  for  flight  management.  In  such  a  case,  the 
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operator  must  divide  his  attention  between  control  and  discrete  tasks. 

An  analytical  model  is  developed  based  on  optimal  control  theory  with 
differential  weight  structure  embedded  in  the  cost  function.  The  model 
performs  the  discrete  control  task  whenever  the  perceived  error  exceeds 
certain  bounds,  as  it  matches  to  the  operator’s  behavior. 

Krishna-Rao,  Ephrath  and  Kleinman  (1979)  have  sought  to  "transform"  the 
optimal  control  model  into  an  optimal  decision  model  that  is  suitable 
for  the  multi-task  situation  where  tasks  of  different  value,  time  re¬ 
quirements,  and  deadlines  compete  for  the  operator's  attention.  The 
model,  bearing  conceptual  similarity  to  the  optimal  control  model,  con¬ 
sists  of  two  separate  blocks:  information-processor  and  decision  proces¬ 
sor.  The  information-processor  block  provides  the  estimates  of  the 
"decision-state,"  i.e.,  the  amount  of  time  required  and  the  amount  of 
time  allowed  to  complete  each  task.  These  estimates,  along  with  task 
values  under  the  subjective  expected  utility  framework  provide  the 
selection  of  a  task  in  the  decision-processor  block.  The  approach  is 
quite  general  and  suitable  for  discrete  task  dynamics. 

Another  approach,  apart  from  general  state-space  control  theory  points- 
of-view,  is  to  start  with  task  analysis  or  task-paradigm  development, 

and  then  to  identify  the  subprocesses  and  the  interactions  among  them. 

1 

Abstraction  of  the  process  dynamics  and  interaction  then  leads  the  way 
toward  simulation  techniques  and  models.  This  approach  is  represented 
by  Seifert  (1979),  Baron  et  al .  (1980),  Tulga  and  Sheridan  (1980), 
Greenstein  (1979),  and  Chu  and  Rouse  (1979). 

Seifert  (1979)  has  proposed  a  combined  SAINT  discrete  network  model  with 
an  optimal  control  model  formulation.  The  objective  is  to  real istical ly 
examine  and  model  an  advanced  man-machine  system  in  which  both  discrete 
tasks  and  continuous  tracking  behaviors  are  exhibited.  The  mutual  in- 
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teraction  between  discrete  tasks  and  continuous  state  variables  is 
achieved  through  "mix-initiation,"  i.e.,  either  by  tasks  being  completed 
or  by  state  variables  crossing  specific  threshold  values.  The  feasibil¬ 
ity  of  employing  this  modeling  approach  has  been  demonstrated  in  a  com¬ 
bined  flight  control  with  multifunction  keyboard  tasks  in  the  Digital 
Avionic  Information  System. 

Baron,  Zacharias,  Mural idharan  and  Lancraft  (1980)  have  studied  flight 
crew  procedures  in  approach  to  landing.  A  simulation  model  based  on 
time-line  analysis  of  nominal  procedures  was  developed.  Their  approach 
draws  heavily  on  the  concepts  and  submodels  of  the  optimal  control  model 
for  the  human  operator.  The  overall  model  structure,  however,  is  one  of 
subprocess  interaction. 

Tulga  and  Sheridan  (1980)  developed  a  multi-task,  supervisory  control 
paradigm  and  a  dynamic  programming  model  of  monitoring  and  control 
behavior.  The  paradigm  addressed  is  one  of  allocating  in  time  a  limited 
attention  resource  to  multiple  simultaneous  demands  of  varying  duration, 
production  rate,  and  rewards.  The  model,  embedding  several  criterion 
functions  and  including  response  time  and  future  discount  constraints, 
allows  one  to  explore  the  interacting  factors  of  task  demands, 
operator's  "plan-ahead"  behavior,  and  task  parameter  estimation  on  the 
performance  of  multiple-process  supervision. 

Greenstein  and  Rouse  (1982)  have  considered  the  operator's  monitoring  of 
multiple  displays  of  stochastic  processes.  A  two-stage  model  that 
represents  the  operator's  event  detection  task  and  attention  allocation 
task  is  developed.  In  the  first  stage,  a  discriminant  analysis  tech¬ 
nique  is  used  to  model  the  operator's  generation  of  probability  esti¬ 
mates  that  events  have  occurred  after  the  ob:ervation  of  display.  In 
the  second  stage,  action  times  and  delay  costs,  along  wit;,  event  proba- 
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bilities,  are  used  in  the  queueing  framework  to  determine  the  order  of 
tasks  to  be  attended. 


Rouse  (1977)  and  Walden  and  Rouse  (1978)  have  modeled  the  pilot  in  a 
multi-task  flight  management  situation  as  a  "server"  in  a  queue  where 
events  are  the  control  and  check-list  procedure  tasks.  The  events  are 
assumed  to  arrive  with  exponentially  distributed  inter-arrival  time,  and 
to  be  serviced  according  to  their  priority.  The  "arrival  rate"  for  con¬ 
trol  tasks  was  measured  from  experimental  results  for  control  activi¬ 
ties,  and  the  service  rate  was  determined  as  a  free  parameter  of  the 
model.  The  model  performance  (mainly  waiting  time  statistics)  matches 
well  with  the  experimental  data. 

Chu  and  Rouse  (1979)  considered  human-operator  interaction  in  the 
multi-task,  flight  management  situation.  Allocation  of  responsibility 
between  operator  and  computer  is  modeled  as  a  control  process  of  the 
queueing  system.  It  was  proposed  and  demonstrated  in  the  study  that  the 
operator's  workload  can  be  maintained  within  an  acceptable  level,  if 
routing  of  information  handling  responsibility  (such  as  check-list  pro¬ 
cedure)  between  operator  and  computer  can  be  achieved  and  adapted  to 
task  demand. 

All  the  above  models  address  the  process  aspect  of  the  information  han¬ 
dling  only.  The  important  issues  of  information  value  and  the  associat¬ 
ed  decision  strategy  in  information  selection,  in  weighing  probable 
payoffs  against  costs  of  acquiring  the  information,  have  not  been  taken 
into  account  in  all  of  the  previous  work  reviewed.  To  compensate  for 
this  deficiency,  a  continuous  information  handling  model  based  on  com¬ 
bined  multi -attribute  utility  and  queueing  model  will  be  described  in 
the  next  section. 
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2.4  A  Time-Continuous  Information  Handling  Model 

Among  the  operator  task  models  of  information  handling  reviewed  in  Sec¬ 
tion  2.3.3,  the  most  relevnt  modeling  concepts  include  that  of  Sander's 
(1964)  visual  sampling  model,  Smallwood's  (1967)  instrument  monitoring 
model,  Carbonell's  (1968)  queueing  model  of  visual  sampling,  Doetsch's 
(1975)  supervisory  flight  control  concept,  DAIS  system  concept  (Aviation 
Week  and  Space  Technology,  1979),  SAINT  network  model  (Kuperman,  et  al . , 
1977),  Greening's  (1978)  crew/cockpit  modeling  survey,  Rouse's  (1978) 
airborne  information  management  and  Cavalli's  (1978)  discrete-time  pilot 
model.  These  studies  point  out  the  emerging  need  for  a  general  model 
for  operator  information  handling  tasks,  that  is  capable  of  representing 
(1)  stochastic  aspects  of  information  processing,  (2)  general  top-down 
system  organization  and  bottom-up  system  synthesis  process  using  in¬ 
tegrated  display  control,  (3)  discrete  events  of  underlying  continuous 
processes,  (4)  parallel  and  serial  operations,  and  (5)  interactive  con¬ 
trol  and  display  systems. 

» 

To  fulfill  these  modeling  requirements,  the  information  flow  concept 
derived  in  this  study  is  based  on  pragmatic  information  value  and  mul¬ 
tiprocessing  organization.  The  former  notion  refers  to  the  assumptions 
of  the  close  interrelationship  between  information  and  decision  and  the 
abilities  of  the  operator  to  quantify  and  compare  information  (Whit- 
temore  and  Yovits,  1973).  The  later  notion  refers  to  the  use  of  analogy 
of  a  time-shared  computer  in  viewing  operator’s  attention  allocation 
among  a  variety  of  tasks  (Johannson  and  Rouse,  1979).  If  one  is  con¬ 
cerned  with  time  performance  and  productivity  (i.e.,  throughput),  the 
queueing  theory  may  be  an  appropriate  formulation. 

The  queueing  theory  framework  assumes  that  the  operator  has  sequentially 
or  randomly  monitored  the  process  and  has  updated  the  estimates  of  event 
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probabilities.  Upon  detecting  or  judging  an  event's  arrival,  the  opera 
tor  then  places  the  event  in  memory  or  in  a  physical  queue  for  atten¬ 
tion.  The  atomic  decisions  actually  involved  in  this  information  han¬ 
dling  task  are  (1)  monitoring  or  attending  to  a  specific  event,  (2) 
selecting  an  event  for  attention,  (3)  selecting  an  information  source, 
(4)  continued  information  sampling/processing,  (5)  selecting  an  alterna 
tive  event  for  action  (if  preemption  is  allowed),  and  (6)  action  selec¬ 
tion.  In  this  study,  decision  (1),  (4),  (5),  and  (6)  will  be  prespeci¬ 
fied  and  decisions  (2)  and  (3)  will  be  the  main  focus  of  the  modeling 
effort.  The  usual  practices  related  to  decisions  (1)  and  (4)  are  to  at 
tend  a  specific  event  if  there  is  one  and  to  continue  information  pro¬ 
cessing  only  when  there  is  an  incorrect  response  in  information  han¬ 
dling.  Therefore,  except  for  decision  (5)  which  is  determined  by  rules 
or  strategies  (e.g.,  preemptive,  nonpreempti ve)  all  the  prespecified  de 
cisions  can  be  incorporated  in  a  set  of  stochastic  functions,  such  as 
probabilities  of  false  alarm,  incorrect  response  and  missed  event;  and 
set  of  probabilities  for  possible  actions  taken. 

2.4.1  Mathematical  Formulation.  In  the  context  of  the  mathematic 
formulation  (Rouse,  1977),  the  assumption  and  approach  to  characterize 
the  information  flow  of  handling  tasks  are  as  follows: 

(1)  The  information  system  possesses  N  independent  state  (or 
feature)  vectors. 

X-j ,  i  =  1,  2,  ....  N  (System  States) 

Yj,  i  =  1,  2,  ...»  N  (Observer  States) 

(1)  The  events  e^  (the  state  or  feature  variations  observed 
which  call  for  information  handling  and  response  activi¬ 
ties,  such  as  check-list  procedures,  fault  procedures,  en- 
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vironment  clearance,  tactical  maneuver,  and  threat  esti¬ 
mates)  arrive  as  independent  stochastic  processes  with  a 
priori  probability  density  function  (pdf)  of: 

U  (*)  =  fi  (xi)*  1  *  1.  2,  ....  N 
where  is  the  arrival  rate  of  event  i 

(1)  The  events  e^  are  perceived  to  have  occurred  after  a  moni¬ 
toring  epoch  with  the  status  display,  with  probability 

P.  ( - ( Y ^. ) *  the  conditional  probability  of  the  event  given 
the  observed  state. 

(2)  The  prior  statistics  of  information  handling  (service)  time 
for  event  i  using  information  j  with  state  observation  Y  is 
given  as: 

9U  Hv)  *  9u  1  ‘  *•  2 . N  „ 

j  =  1 ,  2,  . . . ,  N;  0  =  £  Ij ,  • 

where  g..  (*|Y)  is  the  pdf  of  service  times  t  for 

7  J  5 

observation  Y,  u . .  is  the  mean  service  rate  for  given 

'  J 

event  i  using  information  j . 

2.4.2  Combined  MAU/Queueing  Model  Functions.  Illustrating  the  com¬ 
bined  modeling  approach  of  information  evaluation  and  processing,  Figure 
2-5  shows  the  functional  block  diagram  of  the  MAU  and  queueing  model  for 
the  shared  man-computer  information  handling  task.  The  information  ar¬ 
rivals  are  generated  from  external  information  sources  and  transformed 
into  a  visual  format  (graphic,  schematic,  alphanumeric,  symbol,  or 
tones,  etc.).  Each  new  information  arrival  causes  a  reevaluation  and 
then  a  reformatting  and  a  reordering  of  the  event  queue.  The  computa- 
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FIGURE  2-5. 

FUNCTIONAL  BLOCK  DIAGRAM  OF  MAU/QUEUEING  MODEL 
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tion  of  information  value  if  carried  out  by  the  MAU  model  according  to 
the  set  of  criteria  established  in  the  pervious  section  --  cost,  detection 
and  expected  payoffs/loss,  etc.  The  attribute  levels  and  weights  may  be 
pre-assigned  or  estimated  adpatively  from  the  previous  decision  outcomes. 
Present  development  assumes  a  nonpreemptive  priority  discipline. 

2.4.3  Incorporation  of  Multi-Attribute  Criterion  Function.  The  develop¬ 
ment  to  this  point  has  not  considered  the  criteria  for  the  selection  from 
multiple  information  choices.  In  the  context  of  supervisory  control  of 
advanced  aircraft,  multiple-objective  criterion  function  needs  to  be 
considered.  The  complete  "state  of  nature"  is  assumed  to  be  characterized 
by  the  states  related  to  threat,  navigation,  and  subsystem  situations: 

N  =  <ztUt,  V  zs)> 

where  zt,  zn,  zs  are  time-varying  random  state  vectors. 

The  selection  of  process  or  source  s  and  message  type  j  at  a  particular 
time  is  assumed  to  be  based  on  maximizing  expected  utility 

max  max  [EU(e  .)]  s  =  I,  2,  3,  ...,  S 
si 

j  =  1,  2,  ...,  ms 

where  EU(eSj)  is  the  utility  of  selecting  message  type  j  with  source  s, 

and  EU(eSj)  =  jt  P(0h| s ,  j,  k)  U(0h | s ,  j,  k) 

where  P (0h ) s ,  j,  k)  and  U(0h J  s ,  j,  k)  are  the  probability  and  utility 

of  outcome  0^  given  information  source  s,  message  j,  and  state  k; 
h  =  1,  2,  ... ,  H  •  k  =  1,  2,  K 
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5 

If  we  let  MAU(e$j)  ^  Xijs’  as  su99ested  in  Section  2.3.2,  then  the 
attribute  X..  is  given  by: 

I  J  ^ 

Xijs  '  H  Ps  (0„|j.  k)-*sf(Oh|j.  k) 
h  k 

where  X^is  the  scaled  expected  utility  of  outcome  payoffs  on  given 
information  1^  and  state  z^. 

A  man-machine  simulation  based  on  the  combined  MAU  information  model  and 
queueing  framework  was  developed  for  the  continuous  monitoring  and  control 
situation.  Central  to  this  simulation  is  the  implementation  of  the 
derived  situation  formulation  and  information  selection  criteria  described 
above.  Figure  2-6  provides  a  schematic  diagram  of  the  implementation  of 
model-based  evaluation  procedure.  As  the  diagram  shows,  the  evaluation 
process  can  be  used  in  a  time-continuous  manner,  move  from  one  event  to 
the  next,  one  decision  epoch  to  another,  generate  the  probability  and 
utility  updates  using  the  MAU  model,  select  the  information  sources,  and 
record  actual  system  changes.  Details  of  the  task  simulation  will  be 
described  in  the  next  chapter. 


2-33 


FIGURE  2-6. 

IMPLEMENTATION  OF  MODEL  BASED  EVALUATION  PROCEDURE 
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3.  EXPERIMENTAL  STUDY 


3.1  liyerview 


The  present  experiment  was  designed  to  assess  the  effectiveness  of 
value-based  information  management  schemes  and  the  validity  of  the 
MAU/Queueing  model  in  describing  an  operator's  information  handling 
task.  An  advanced  aircraft  simulation  of  a  multi-task  situation  was 
developed.  Individual  subjects  were  required  to  pilot  a  simulated  air¬ 
craft  in  a  changing,  hostile  environment.  In  doing  so,  they  were  pro¬ 
vided  with  various  forms  of  information  concerning  the  multiple  threats 
encountered  and  subsequent  evasive  or  aggressive  actions.  In  addition, 
the  subjects  were  required  to  monitor  subsystem  functions  and  to  perform 
checklist  procedures.  Performance  comparisons  were  made  in  the  study 
levels  of  computer  aiding  and  between  levels  of  task  demands. 

3.2  Hypotheses 

The  following  experimental  hypotheses  were  .tested: 

(1)  A  combined  queueing/mul ti -attribute  utility  model  is  suit¬ 
able  for  representing  various  information  handling  opera¬ 
tions  in  advanced  aircraft.  The  multi -attribute  model  is 
useful  for  evaluating  the  effectiveness  of  different  infor¬ 
mation  system  configurations  under  a  variety  of  task  condi¬ 
tions.  The  queueing  model  is  useful  for  describing  infor¬ 
mation  handling  characteristics  (such  as  demand,  procedure, 
and  performance)  for  multiple  information  situations  and 
for  a  variety  of  task  conditions  and  computer  aiding.  The 
combined  model  is  capable  of  depicting  the  operator's 
preference  in  information  selection. 
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(2)  The  improvement  of  aided  performance  under  option  recommen¬ 
dation  and  automatic  selection  will  be  enhanced  in  condi¬ 
tions  of  high  information  load. 

3.3  Task  Simulation 


Information  requirements  and  operator  tasks  in  advanced  aircraft  opera¬ 
tions  were  presented  in  last  year's  report  (Chu,  Steeb,  Freedy,  1980). 
The  operator's  functions  include  navigation,  communication,  and  monitor¬ 
ing  enemy  activity;  searching,  acquiring  and  tracking;  delivering 
weapon;  and  evading  enemy  action.  The  operator’s  tasks  include:  selec¬ 
tion  of  equipment/mode,  monitoring  and  observing;  visual  search,  ac¬ 
quisition  and  tracking;  monitoring  and  following  command,  etc.  Both 
sequential  and  parallel  information  handling  are  required.  In  the  task 
simulation  developed,  the  parallel  and  independent  handling  requirements 
were  represented  by  a  multiple-process  check-list  monitoring  and  execu- 
tional  procedure.  The  sequential  information  handling  requirements  were 
represented  by  a  three-stage  (search-selection-action)  threat-intercept 
operation.  In  addition,  under  the  categories  of  computer  aiding  (aggre¬ 
gation,  alerting,  and  option  recommendation),  the  operation  procedures 
presented  the  possibility  of  various  levels  of  interaction  requirements. 
Overall,  the  task  situation  was  a  multi-task  one,  with  mixed  levels  of 
computer  aiding. 

The  task  environment  chosen  was  an  adaptation  and  extension  of  the  mul¬ 
tiple  threat  intercept  simulation  employed  in  the  previous  program 
(Steeb,  Chu,  Clark,  Alperovitch,  Freedy,  1979).  The  symbols  and  types 
of  threat  information  were  similar  to  that  used  previously,  but  the 
display  format  and  threat  occurrence  were  varied,  encompassing  time- 
varying  multiple-task  situation.  Consequently,  a  wider  variety  of  tasks 
and  information  handling  options  were  available  to  the  operator.  Multi- 
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pie  task  demands,  including  (a)  threats  of  uncertain  capability  and  lo¬ 
cation,  and  (b)  subsystem  events  of  sporadic  occurrence  but  with  a 
structured  procedure  arose  continuously.  The  operator  had  the  option  of 
selecting  a  particular  event  to  attend  to  and  the  option  of  accessing  a 
particular  source  of  information.  The  characteristics  of  the  event  dif¬ 
fered  in  payoff,  urgency,  and  delay.  The  forms  of  information  differed 
in  threat  discrimination  capabilities,  transmission  costs,  processing 
delays,  and  potential  of  detection.  The  simulated  task  environment  is 
shown  in  Figure  3-1  and  will  be  discussed  in  the  following  subsections. 

3.3.1  Category  of  Task  Demands.  As  described  above,  there  are  two 
categories  of  task  demands:  high  priority  and  low-priority.  The  high- 
priority  task  was  the  threat  intercept  task  of  navigating  a  simulated 
aircraft  through  a  changing,  hazardous  environment.  The  environment 
contained  threats  of  uncertain  form  and  location,  and  the  identity  and 
location  of  the  threat  could  be  determined  by  requesting  different  types 
of  information.  The  different  types  of  information  varied  in  cost  and 
content,  so  that,  at  each  threat,  the  subject  either  took  an  avoidance 
action  or  an  aggressive  action.  The  low  priority  task  monitored  the 
subsystem  displays  that  simulated  status  indicators  of  aircraft  func¬ 
tions,  and  processed  interactive  check-list  procedures  for  the  correc¬ 
tion  of  malfunctions.  The  two  categories  of  tasks  are  discussed  in  the 
following  paragraphs. 

Threat  Intercept  Task.  The  threat  environment  and  the  vehicle  were 
shown  in  a  moving-map  display,  where  incoming  threats  appeared  at  ran¬ 
dom,  presented  with  symbol  the  "D"  at  the  upper  edge  of  the  screen,  and 
moved  downward  at  a  constant  velocity.  The  operator  could  move  the 
vehicle  symbol  horizontally  to  avoid  the  obstacles,  or  to  fire  at  the 
nearest  threat. 
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The  threats  introduce  both  uncertainty  and  danger  to  the  task  simula¬ 
tion,  and  each  type  of  threat  has  a  region  of  possible  damage  to  the 
aircraft  as  shown  in  Figure  3-2.  The  probability  of  damage  is  a  func¬ 
tion  of  the  horizontal  distance  between  the  threat  and  the  piloted  air¬ 
craft.  For  ease  of  learning,  the  four  obstacle  types  are  designed  to  be 
evocative  of  the  types  of  contact  expected  to  occur  in  actual  flight 
missions  --  missile,  fixed-wing  aircraft,  helicopters,  and  false  alarms 
(bi rds) . 

Threat  information  was  categorized  into  six  types:  Full,  Outline,  Bio¬ 
logical,  Location,  Left/Right,  and  Default.  The  corresponding  type  and 
location  di scriminabi 1 ity  and  symbols  used  are  given  in  Table  3-1.  In 
such  an  environment,  a  typical  sequence  of  operations  in  threat  informa¬ 
tion  handling  is  as  follows: 

(1)  Event  detection  --  when  a  "D"  symbol  appears  and  moves  down 
the  screen. 

(2)  Event  selection  --  when  both  threat  and  subsystem  events 
are  present. 

(3)  Information  seeking  --  where  various  types  of  information 
are  evaluated  and  the  selection  of  detailed  information  is 
performed. 


M-Missile 
A-Aircraft 
H-Hel i copter 
B-Bird 


CORRIDOR  LOCATION 


FIGURE  3-2. 

THREAT  CHARACTERISTICS 


TABLE  3-1 


INFORMATION  SOURCE  CHARACTERISTICS 


Information  Type 

Discrimination 

Location 

Symbc 

Full 

All 

All 

M,  A,  1 

Outl ine 

All  but  Airplane/ 
He! i copter 

All 

M,  AH, 

Biological 

Bird  Only 

All 

MAH,  B 

Location 

None 

All 

X 

Left/Right 

All 

Left/Right 

M,  A,  1 

Defaul t 

None 

None 

D 
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(4)  Information  confirmation  --  observe  and  interpret  the  sym¬ 
bol  displayed  on  the  screen  when  both  the  time  delay  and 
information  content  are  varied  for  different  information 
types  selected. 

(5)  Action  decision  --  aggression  or  avoidance. 

Subsystem  Monitoring  Tasks.  The  subsystem  task  required  the  operator  to 
monitor  various  levels  of  the  subsystem  processes  (Figure  3-3)  to  detect 
possible  events  (i.e.,  "0"  symbols)  and  to  act  to  bring  the  process  into 
operational  status  (i.e.,  "1"  symbols).  If  the  operator  thought  process 
b  were  down  (Figure  3-3A),  he  might  press  "6"  on  the  keypad  and  the 

status  display  of  the  next  level  would  appear  (Figure  3-3B)  The  display 

might  then  show  that  branch  3  were  "down"  and  a  press  of  key  "3"  would 
lead  to  the  display  of  next  level,  where  the  branch  "4"  would  be  "down," 
(Figure  3-3C)  etc.  The  operator  would  continue  until  all  branches  were 
"up",  whereupon  process  6  would  be  "up"  again. 

Speed  and  errors  were  recorded  on  performance  data.  When  the  incorrect 
key  was  pressed,  an  error  was  registered  in  the  system  while  the  display 
status  remained  unchanged. 

3.3.2  Displays  and  Controls.  The  simulation  used  a  computer- 
generated  graphic  display  and  an  alphanumeric  CRT  display  illustrated  in 
Figure  3-1.  The  display  on  the  left  was  the  graphic  display  of  the 

threat  handling  tasks,  for  which  the  subjects  had  to  select  the  ap¬ 

propriate  information  for'the  follow-on  actions,  evasive  or  aggressive. 
The  display  on  the  right  was  of  the  operator's  monitoring  tasks,  for 
which  the  subjects  were  required  to  attend  to  the  interactive  check-list 
procedures  for  subsystem  abnormalities  and  to  maintain  the  system  under 
operational  conditions. 
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State  1  1  ...  0 


Figure  3 

-3A.  Monitoring  Display 

^  Proc 

6  /  Level 

1 
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6 
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1 
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Figure  3-3B. 

First  Level  of 

Process 

6 

^  Proc 

6  /  Level 

2 

Branch 

12  3  4 

•  * 

6 

State 

1110 

•  • 

1 

Figure  3-3C.  Second  Level  of  Process  6 

FIGURE  3-3. 

SUBSYSTEM  CHECKLIST  PROCEDURE 


The  real-time  graphics  used  in  the  study  provided  the  fidelity  in  timel¬ 
iness  and  sensor  data  variability.  Color  was  used  to  differentiate 
"dynamic"  foreground  information  from  the  "static"  background  reference. 
There  were  three  functional  elements  in  the  graphic  display  --  the  mov¬ 
ing  map  display,  the  situational  display,  and  the  outcome  display.  They 
are  described  in  the  following  paragraphs. 

Moving  Map  Display.  Spatial  relationships  of  the  aircraft  relative  to 
the  sensed  threat  environment  were  presented  in  the  upper-right  portion 
of  the  graphic  display  (Figure  3-4).  Sensed  target  types  and  the  ap¬ 
proximated  position  (foreground)  were  shown  as  target  symbols  in  a  fixed 
assessment  area  and  acquisition  window  (background).  The  display  was 
updated  with  inputs  from  information  selection  keys,  own  vehicle  (joys¬ 
tick)  control,  and  detected  threat  arrivals.  The  own  vehicle  could  be 
moved  horizontally  within  the  window,  either  to  avoid  the  threats  or 
take  an  aggressive  action  against  one  of  the  threats.  The  actions  open 
to  the  operator  were  primarily  decision  making  in  nature,  the  dynamics 
of  control  were  minimized  since  the  threat  and  vehicle  velocities  were 
held  constant. 

Situation  Display.  The  stages  of  an  aircraft  mission  can  be  character¬ 
ized  by  such  factors  as  danger,  difficulty,  system  reliability,  and  com¬ 
munications  security.  These  factors  vary  from  situation  to  situation 
and  have  consistently  been  the  primary  consideration  in  airborne  infor¬ 
mation  handling.  In  this  study,  the  situational  conditions  that  vary 
among  experimental  phases  were: 

(1)  Degree  of  danger  -  this  was  the  distribution  of  possible 
threats  in  a  given  phase.  A  different  set  of  probabilities 
of  the  four  threat  types  was  assigned  to  each  phase. 
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TARGET 


FIGURE  3-4. 

MAP,  SITUATION  AND  OUTCOME  DISPLAYS 
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(2)  Information  accuracy  -  a  percentage  of  information 
transmissions  that  were  inaccurate  through  having  random 
noise  added  to  location  content.  This  indicated  the  possi¬ 
bility  of  the  location  being  unpredictable. 

(3)  Payoffs  -  different  payoffs  in  points  were  made  for 
avoidance  or  damage  sustained  from  the  threats,  and/or  for 
successful  or  unsuccessful  aggression  actions  toward  the 
threats.  The  payoffs  for  successful  and  unsuccessful  ac¬ 
tions  were  presented  by  a  series  of  numbers. 

The  situational  conditions  that  vary  from  decision  to  decision  were: 

(1)  Costs  -  a  different  and  time-varying  cost  was  assigned  to 
each  information  choice.  This  was  represented  as  the 
number  of  points  that  the  information  would  cost  (by  sub¬ 
tracting  from  the  possible  payoff). 

(2)  Delay  -  the  delay  in  half  seconds  was  presented  regarding 
the  time  required  for  acquiring  the  selected  information 
type.  This  was  equivalent  to  the  portion  of  the  distance 
the  default  symbol  traveled  after  the  information  request 
and  before  the  "true"  target  symbol  was  displayed. 

(3)  Detection  -  this  was  the  increased  danger  on  the  succeeding 
decision  due  to  the  use  of  a  given  information  source.  A 
.10  detection  projected  that  a  damage  would  increase  by  10% 
on  next  decision. 


Outcome  Display.  The  element  on  the  bottom  of  the  graphic  display 
represented  the  immediate  feedback  to  the  subjects  concerning  the  out- 


come  and  performance  score  resulting  from  their  decisions.  The  outcome 
display  was  necessary  due  to  the  probabilistic  nature  of  the  true  state 
and  also  for  motivational  purposes.  The  display  consisted  of  two  parts 
the  outcome  message  and  the  performance  score.  The  outcome  message  in¬ 
cludes  ‘avoid1  or  'damage'  for  avoidance  action;  and  'hit  and  avoid', 
'miss  and  avoid',  or  'miss  and  damage'  for  aggressive  action.  The  per¬ 
formance  score  was  given  as  an  updated  number  based  on  the  payoffs  and 
cost  incurred  in  each  threat  intercept.  The  characteristics  of  each 
functional  display  along  with  the  subsystem  display  are  summarized  in 
Table  3-2. 

3.4  Information  Handling  Model 


3.4.1  General .  The  continued  information  handling  situation  faced  by 
the  operator  is  a  multiple-stage  information-action  sequence,  as  shown 
in  Figure  3-5.  The  assumption  is  that  the  operator  sequentially  sampled 
different  sources  until  some  confidence  level  was  achieved  prior  to  ac¬ 
tion  execution.  Hence,  the  experimental  scenario  of  combined  threat  in¬ 
tercept,  navigation,  and  flight  management  operations  can  be  represented 
in  a  cascade  of  decision  tree  structures.  Specifically,  the  threat  in¬ 
formation  handling  task  has  been  formulated  in  detailed  and  explicit 
selection  processes,  as  shown  in  Figure  3-6.  The  decision  space  is 
fairly  large  resulting  from  the  six  possible  information  choices,  four¬ 
teen  message  types,  two  subsequent  action  choices,  continuous  range  of 
threat  position,  and  24  combinations  of  threat  status.  A  variety  of 
time  varying,  multi -dimensional  consequences  result  from  the  various 
combinations  of  navigation  states,  outcomes,  costs,  payoffs,  delays  and 
future  impacts. 

A  purely  analytical  formulation  of  this  is  intractable,  just  as  it  is 
for  most  operational  information  seeking  decisions.  Categorization  is 


TABLE  3-2 


DISPLAY-CONTROL  CHARACTERISTICS 


DISPLAY  TYPE 

FUNCTION 

CONTROL 

ACTIVE  ELEMENTS 

PASSIVE  ELEMENTS 

UPDATE  EVENTS 

MOVING-MAP 

Threat  - 
Own  Vehicle 

Relations 

Information/Action 
Selection  Keys, 
Joystick 

Threat  ID, 

Threat  Position 

Vehicle  Position, 
and  Direction 

Assessment  Area 
Assessment  Window 

Joystick  Control, 

Threat  Arrival , 
Information  Selection, 
Action  Selection 

SITUATION 

Information 

Options  and 

Handl Ing 

Attributes 

Action 

Selection 

Keys 

Estimates  for 

Cost,  Delay, 
and  Detection 

Estimates  for 

Accuracy,  Payoffs 
and  Degree  of 

Danger 

Action  Selection 

Phase  Transition 

OUTCOME 

Feedback  on 
Selectlon/Actlon 

Outcome  and 

Score 

Action 

Selection 

Keys,  Joystick 

Hit/Miss, 

Avoid/Damage , 

Score 

Anchors 

Action  Selection 

SUBSYSTEM 

Operation  and 
Check-Point 

Status  of 

Subsystems 

Alphanumeric 

Keys 

Levels 
of  Status 

Anchors 

Subsystem  Events 
Checking 

Procedures 
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an  obvious  means  of  reducing  the  complexity  of  the  decision.  Here  those 
elements  in  the  decision  similar  in  consequence  can  be  classified  to¬ 
gether.  For  example,  the  navigational  area  can  be  lumped  into  22 
discrete  regions  according  to  the  hit  and  the  avoidance  probabilities, 
and  the  number  of  threat  states  can  be  reduced  to  four  categories  by  ig¬ 
noring  specific  locations.  The  probability  of  each  message  and  state 
can  then  be  calculated  from  the  prior  probabilities  of  each  of  the 
threats  and  the  information  source  characteristics.  The  actions  can  be 
similarly  categorized  as  avoid  or  attack  without  regard  to  location. 
Probabilities  of  each  outcome  type--avoidance ,  damage,  miss  and 
avoidance,  miss  and  damage,  and  hit--can  be  established  by  observed  fre¬ 
quency.  To  do  so,  a  probability  estimate  must  be  associated  with  each 
combination  of  information,  message,  action,  and  state.  After  categori¬ 
zation,  48  such  combinations  are  present.  These  probabilities  were 
determined  from  a  series  of  pilot  system  tests,  and  were  intended  to  be 
representative  of  the  performance  of  the  typical  subject.  Estimates 
specific  to  each  subject  were  not  made.  At  a  given  time,  the  conse¬ 
quence  levels  (the  attribute  level  vector)  associated  with  a  given  in¬ 
formation  choice  in  a  given  situation  are  calculated  by  folding  back  the 
decision  tree.  The  favored  action  choice  after  receipt  of  a  given  mes¬ 
sage  is  determined  in  a  similar  manner. 

3.4.2  Threat  Information  Handling  Attributes.  Five  time-varying, 
consequence-related  attributes  were  employed  in  the  decision  model. 

The  attributes  are  the  following: 

(1)  (Xj)  Cost  -  The  cost  of  the  communication  in  points  (costs 
ranged  from  0  to  14.5  points). 

(2)  (X2)  Delay  -  The  time  in  seconds  before  display  of  the  in¬ 
formation  (delays  ranged  from  0  to  4  seconds). 
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(3)  (X3)  Detection  -  Increase  in  the  probability  of  damage  on 
the  subsequent  decision  (the  probability  of  increase  ranged 
from  1  to  5  percent). 

(4)  (X4)  Vehicle  loss  -  Expected  (probability  weighted)  level 
of  avoidance  damage  to  own  vehicle. 

(5)  (X5)  Offensive  gain  -  Expected  level  of  damage  inflicted  on 
adversary . 

X4  and  X5  are  computed  according  to  the  following  expression: 

M  state 

X.  (Avoid  |  Mes.,  Info.)  =  £  [P(State J  Mes, ,  Info. )* 

J  fc=l  1  3  1 

X4(State2  j  MeSj,  Info.)] 


M  state 

X5 (Attack  |  Mes.,  Info.)  =  £  [P($tate.  |  Mes.,  Info.)* 
J  £=i  1  J  1 

X5(StateJ  MeSj ,  Info.)] 


where 

X4(StateJ Act^,  MeSj,  Info..)  =  P(Avoid,  StateJ  Act^,  MeSj,  Info^* 

Payoff  (Phase,  Avoidance)  +  {l-P(Avoid,  StatejAct^,  Mes^,  Info^}* 

Payoff  (Phase,  Damage) 
and 

X^StateJ Act^,  Mes^,  Info^  =  P(Hit,  StatejActk,  Mes ^ ,  Info..)* 
Payoff  (Phase,  Hit)  +  {l-P(Hit,  State  |Actt,  Mes.,  Info.)}* 

Payoff  (Phase,  Miss). 
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The  payoffs  range  from  -14  to  14  points. 


3.4.3  Option  Evaluation.  The  evaluation  of  each  of  the  6  information 
choice  is  made  according  to  the  following  equation: 

5 

MAU  [I$]  l  K.  -Xs 
i  =  l 


where  MAU  [Is]  is  the  aggregate  (multi-attribute)  utility  of  information 
choice  I$ ,  X.-s  is  the  level  of  attribute  i  associated  with  information 
choice  Is ,  (calculated  using  Equation  2-3)  and  Kn-  is  the  importance 
weight  of  attribute  i.  It  should  be  noted  that  the  program  did  not  have 
access  to  the  true  state  of  the  environment. 

Adaptive  estimation  of  importance  weights  was  employed  in  the  study,  us¬ 
ing  the  pattern  recognition  method  described  in  Section  2.3.  Since  tri¬ 
al  experiences  demonstrated  that  subjects  were  not  able  to  produce  dif¬ 
ferent  weight  vectors  for  each  phase,  but  rather  followed  a  single 
overall  strategy,  a  5-element  vector  was  maintained  for  each  of  the 
three  phases. 

3.5  Experimental  Procedure 


3.5.1  Experimental  Variables.  The  following  experimental  variables 
and  levels  were  tested. 

(1)  Computer  aiding  -  three  levels. 
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(a)  Unaided  operation  -  the  operator  makes  the  information 
and  control  choices  without  benefit  of  aiding. 


(b)  Situation  aggregation  -  the  operator  makes  the  infor¬ 
mation  and  control  choices  with  the  aids  of  aggregated 
situational  information  and  display. 

(c)  Information  recommendation  and  automated  selection  - 
the  operator  makes  the  information  and  control  choices 
with  the  aids  of  computer  recommendations,  with  the 
option  to  override. 

(2)  Threat  arrival  rate  -  two  levels. 

(a)  Low  speed  stress  -  threats,  including  false-alarm,  ar¬ 
rived  with  an  average  interarrival  time  of  30  sec. 

(b)  High  speed  stress  -  threats  arrived  with  an  average 

interarrival  time  of  15  sec.  , 

The  low  and  high  speed  stress  levels  were  chosen  empirically  to 
represent  two  reasonable  extremes  of  load.  The  low  speed  rate  was 
selected  to  provide  sufficient  time  for  trade-off  consideration  of  all 
factors.  The  high  speed  rate  was  designed  to  rush  the  information  and 
action  decision  somewhat,  but  not  to  debilitate  the  action  accuracy. 

3.5.2  Subject  and  procedure.  An  experiment  based  on  the  representa¬ 
tion  described  above  was  conducted.  First,  six  subjects  were  used  in  a 
preliminary  experiment.  Those  data  were  used  to  adjust  the  situational 
parameters.  Another  twelve  subjects  were  recruited  from  nearby  univer¬ 
sities  for  the  actual  experiment.  All  subjects  represented  the  type  of 
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personnel  who  might  interface  with  computer-aided  information  systems. 

The  subject's  ages  ranged  from  18  to  30.  All  had  one  or  more  years  of 
college  experience.  Six  were  male  and  six  were  female.  Four  subjects 
had  had  experience  with  computers.  The  twelve  subjects  were  assigned 
randomly  to  the  six  groups. 

Each  subject  underwent  two  hours  of  orientation  and  practice  of  both 
subsystem  monitoring  and  threat  intercept  tasks.  The  practice  concluded 
with  a  high  arrival  rate  manual  session,  during  which  aggregated  attri¬ 
bute  information  was  given.  The  experimental  sessions  consisted  of 
three  complete  phases  of  cruise,  surveillance  and  aggression  phases,  ten 
minutes  in  each  phase.  For  each  experimental  session,  the  subject  was 
first  told  the  specific  tasks  to  perform,  then  a  30-minute  trial  was 
given,  and  a  questionnaire  (in  the  form  that  is  shown  in  Appendix  III)  was 
filled  out  by  the  subject.  Each  subject  experienced  all  six  combina¬ 
tions  of  conditions  in  a  repeated  measures  design  (Table  3-3).  The  sub¬ 
jects  were  paid  $6.00  per  hour  and  were  given  a  bonus  of  up  to  $6.00  per 
hour  contingent  on  performance. 

3.5.3  Performance  Measures.  The  following  performance  measures  were 
evaluated  in  every  experimental  run: 

(1)  Threat  handling  task  score.  The  score  was  derived  from 
payoffs/penalties  and  communication  costs.  The  score  was 
presented  to  the  subject  as  a  single  index  of  performance, 
and  the  subject's  compensation  depends  to  a  large  extent  on 
this  measure. 

(2)  Average  delay  in  response  and  service  for  subsystem  events. 
The  subsystem  event  response  time  was  measured  from  the 
time  of  event  occurrence  to  the  time  at  which  an  action  was 
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TABLE  3-3 


EXPERIMENTAL  DESIGN 


LOW  SPEED  STRESS 

HIGH  SPEED  STRESS 

MANUAL 

NO  UPDATE 

MANUAL 

UPDATED 

AUTO 

MANUAL 

NO  UPDATE 

MANUAL 
' UPDATED 

AUTO 

1 

1 

6 

4 

5 

3 

2 

2 

2 

3 

5 

4 

6 

1 

Q_ 

O 

3 

3 

5 

1 

2 

4 

6 

a: 

o 

4 

4 

2 

6 

3 

1 

5 

5 

5 

1 

3 

6 

2 

4 

6 

6 

4 

2 

1 

5 

3 

Numbers  denote  sequence  of  presentation  of  conditions. 
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initiated.  The  service  time  was  measured  from  the  time  of 
last  action  initiation  to  the  time  of  action  completion  for 
the  event.  The  waiting  time  was  measured  from  the  time  of 
event  occurrence  to  the  time  of  action  completion  for  the 
event.  Waiting  time  was  equal  to  the  sum  of  response  time 
and  service  time  only  when  non-preempti ve  or  preemptive- 
resume  disciplines  were  used  and  no  incorrect  action  oc¬ 
curred  . 

(3)  Subsystem  service  errors,  i.e.,  incorrect  actions.  As  the 
probabilities  of  both  false  alarms  and  missed  events  were 
low  and  were  not  recorded,  the  subsystem  service  errors 
reerred  to  the  incorrect  keys  pressed. 

(4)  Information  selection  and  action  times  for  threat  handling 
task. 

(5)  Operator  time  occupancy  in  terms  of  the  fraction  of  time 
the  operator  was  performing  either  subsystem  or  threat  han- 
dl ing  tasks. 

(6)  Subjective  ratings  of  level  of  effort  required  for  the 
tasks  and  the  desirability  of  computer  aiding. 

All  these  measures  were  obtained  by  analyzing  the  sampled  and  the  accu¬ 
mulated  data.  Except  for  subsystem  status  and  aircraft  stick  responses, 
which  were  sampled  twice  per  second,  all  data  were  sampled  as. nchronous- 
ly.  The  empirical  results  along  with  the  analysis  of  variance  are  dis¬ 
cussed  in  the  next  chapter. 


3-23 


4.  EXPERIMENTAL  RESULTS  AND  DISCUSSION 


4.1  Overview 


This  chapter  summarizes  the  results  of  the  experiment  performed  in  the 
current  program  year  to  investigate  the  efficacy  of  model -based  aiding 
in  information  handling.  The  experimental  situation  was  sufficiently 
varied  and  difficult  to  provide  a  test  of  the  MAU/Queueing  model.  The 
subject  learned  the  task  procedures  readily,  and  by  the  end  of  the 
training  session,  could  effectively  handle  the  multiple  task  demands 
under  different  aiding  conditions.  A  variety  of  subject  preferences  and 
strategies  were  observed  and  modeled. 

The  data  sampled  during  the  flight  information  handling  experiment  was 
analyzed  to  obtain  the  several  objective  measures  listed  in  the  previous 
chapter.  The  subjective  ratings  of  the  task  situations  based  on  the 
questionnaires  answered  by  the  subjects  during  the  experiment  were  also 
obtained.  For  each  of  these  measures,  factors  of  significance  were 
determined  using  the  analysis  of  variance  (fixed  effects,  within  subject 
design)  and  the  underlying  trends  of  variation  were  investigated  (ANOVA 
tables  appear  in  Appendix  IV).  Effects  were  accepted  as  significance 
if  p  <  0.05. 

4.2  Task  Performance 


4.2.1  Task  Score.  Figure  4-1  shows  the  performance  score  attained, 
averaged  across  the  subjects  for  the  threat  intercept  task  under  three 
modes  of  computer  aiding  and  the  two  levels  of  threat  arrival  rate. 
Both  experimental  variables  produced  statistically  significant  effects 
[F(2,  22)  =  112.10,  p  <  0.001  for  aiding  mode;  F(l,  11)  =  473.04,  p  < 
0.001  for  arrival  rate].  The  subjects’  performance  score  increased  as 
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the  computer  aiding  level  increased  and  as  threat  arrival  rate  in¬ 
creased.  Model  recommendation  (REC)  mode  has  maintained  a  60%  improve¬ 
ment  and  information  aggregation  (AGGR)  mode  a  25%  improvement  over 
manual  (no  aiding)  mode  for  both  the  low-  and  high-threat-rate  condi¬ 
tions. 

The  score  improvements  with  aggregation  and  recommendation  aids  were 
traceable  to  differences  in  both  action  outcome  and  payoffs/cost  at¬ 
tained.  Subjects  in  both  the  recommendation  and  aggregation  aided  con¬ 
ditions  achieved  more  "avoidance"  and  “hits"  than  in  the  manual,  unaided 
conditions.  Although  the  aided  conditions  incurred  greater  information 
acquisition  cost  than  unaided  conditions,  the  aided  conditions  resulted 
in  higher  payoff  scores.  The  increase  in  payoffs  was  more  than  double 
the  increase  in  cost  expended,  resulting  in  the  net  performance  improve¬ 
ment  observed  in  the  aided  conditions. 

4.2.2  Subsystem  Waiting  Time  and  Service  Errors.  The  average  subsys¬ 
tem  waiting  time,  shown  in  Figure  4-2,  appears  to  be  independent  of  aid¬ 
ing  level,  but  increased  as  threat  arrival  rate  increased  [F(l,  11)  = 
9.51,  p  <  0.01].  This  may  be  due  to  the  subject's  preemption  of  subsys¬ 
tem  service  by  the  threat  arrivals.  The  service  errors,  measured  as  the 
ratio  of  the  number  of  incorrect  actions  to  the  total  number  of  actions, 
appear  to  be  independent  of  both  arrival  rate  and  aiding  level. 

4.2.3  Subjective  Ratings.  Subjects'  ratings  concerning  the  perceived 
level  of  effort  in  performing  the  tasks,  the  effectiveness  and  the 
desirability  of  computer  aiding,  and  the  ease  of  interaction  with  the 
aiding  were  analyzed.  Individual  ratings  for  different  task  situations 
were  first  converted  to  a  normalized  scale  (e.g.,  Guilford,  1954),  then 
these  measures  of  variation  among  tasks  were  averaged  across  the  sub¬ 
jects.  The  resulting  effort  ratings,  i.e.,  the  averaged  level  of  per- 
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AVERAGE  SUBSYSTEM  WAITING  TIME 


ceived  effort,  varied  from  ‘low*  to  'high1  level  (Figure  4-3),  and  in¬ 
creased  as  threat  arrival  rate  increased  [F(l,  11)  =  29.53].  Although 
both  of  the  aiding  modes  have  been  shown  to  produce  higher  effort  level, 
the  effect  was  not  significant.  It  appears  that  the  subjects  had  actu¬ 
ally  settled  for  lower  performance  and  lower  effort  levels  in  unaided 
situations  than  those  in  aided  situations. 

The  subjective  ratings  of  the  various  aspects  of  computer  aiding  also 
appear  to  vary  among  the  task  conditions.  The  aiding  was  considered 
'relatively  easy'  to  interact  with  (Figure  4-4),  'somewhat  desirable'  by 
the  subjects  (Figure  4-5),  and  had  a  'slight  improvement'  on  performance 
(Figure  4-6).  Among  the  factors  of  significance,  the  subjects  saw  the 
aiding  to  be  relatively  more  effective  [F(2,  22)  =  5.27,  p  <  0.05]  and 
more  desirable  [F(l,ll)  =  56.57,  p  <  0.001]  in  aggregation  mode,  and  re¬ 
latively  easier  to  use  in  high  threat  rate  situations  [F ( 1 ,  11)  =  8.83, 
p  <  0.05).  It  is  interesting  to  note  that  the  subjects'  perceived 
desirability  of  the  aiding  was  more  in  line  with  their  perception  of 
aiding  effectiveness  than  with  their  perception  of  the  ease  of  interac¬ 
tion. 

In  general,  based  on  the  comment  from  the  subjects,  it  appears  that  all 
subjects  were  quite  in  favor  of  both  aiding  schemes  used  in  the  experi¬ 
mental  situation.  They  have  preferred  the  aggregation  aid  to  the  recom¬ 
mendation  aid,  mainly  because  it  took  time  and  effort  to  override  the 
recommended  decisions  however  infrequently  was  the  override. 

4.3  Information  Selection  Behavior 

Each  of  three  aiding  modes  --  manual,  aggregation,  and  recommendation  -- 
exhibited  a  different  distribution  of  selections  among  the  six  informa¬ 
tion  options.  Figure  4-7  shows  the  frequency  of  selection  averaged 
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across  the  subjects.  The  aided  (aggregation  and  recommendation)  selec¬ 
tions  exhibited  more  diverse  use  of  high-detail /high-cost  information 
such  as  Full  and  Outline  information  than  unaided  selections.  The  lack 
of  use  of  the  high-detai 1 /high-cost  information  in  the  unaided  selec¬ 
tions  may  be  due  to  the  subjective  emphasis  on  some  sub-optimal  objec¬ 
tives. 

Subjects'  policy  in  information  selection  can  be  observed  via  the  aver¬ 
aged  importance  weights  they  placed  on  the  attribute  vector,  i.e.,  the 
cost,  delay,  detection,  loss  and  gain  factors.  The  averaged  policies  of 
the  subjects  in  three  aiding  modes  are  shown  in  Figure  4-8.  While  the 
profiles  of  the  three  modes  are  quite  similar,  the  subjects  in  aided 
modes  appeared  to  place  a  greater  emphasis  on  cost  (Aj)  and  aggresive 
gain  ( A^)  than  they  did  in  the  unaided  modes.  The  averaged  policies  of 
the  subjects  in  two  levels  of  threat  arrival  rate  are  also  shown  in  Fig¬ 
ure  4-9.  The  subjects  in  low  stress  situations  appeared  to  place  a 
greater  emphasis  on  communication  cost,  (Aj),  time  delay  (A^)  an d  ag¬ 
gresive  gain  (A3)  than  they  did  in  the  high  stress  situations. 

In  order  to  ascertain  the  effect  of  the  aiding  mode,  speed  stress  and 
tactical  phase  on  the  subject's  policy,  a  multivariate  analysis  of  vari¬ 
ance  was  performed.  The  Pillais  multivariate  test  for  significance 
showed  a  significant  effect  of  mode  by  speed  by  phase  (df.  4,198,  approx, 
f  =  1.8,  p  <  0.015).  Examination  of  the  univariate  f  tests  indicate 
that  the  greatest  effect  was  with  cost  (df.  4,198,  F  =  6.18,  p  <  .0001) 

and  time  delay  (df.  4,198,  F  3  2.98,  p  <  .02),  which  substantiates  the 
observation  that  the  subjects  had  placed  extra  emphasis  on  these  two 
factors.  Multivariate  analyses  were  also  performed  to  examine  the  ef¬ 
fects  of  specific  combinations  of  mode,  speed  and  phase.  These  tests 
show  significant  differences  in  policy  among  aiding  modes,  tactical 
phases,  and  between  stress  conditions.  (MAN0VA  tables  appear  in  Appen¬ 
dix  IV.) 
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5.  CONCLUSIONS  AND  RECOMMENDATIONS 


5. 1  Value-Based  Model  of  Information  Handling  Tasks 


The  present  study  demonstrates  the  potential  of  on-line,  value-based 
modeling  techniques  for  information  source  selection  in  continuous  in¬ 
formation  handling  tasks.  The  techniques  employ  the  multi -attribute 
utility  model  to  represent  the  prescriptive  aspect  and  the  queueing 
model  to  represent  the  descriptive  aspect  of  the  information  handling 
situation.  The  current  year  study  has  used  the  queueing  model  as  a 
framework  to  evaluate  the  value-based  model  in  a  continuous  decision  and 
control  environment,  where  the  operator  is  required  to  make  complex  and 
timely  decisions  regarding  information  and  control  options.  The  model 
was  shown  to  be  useful  for  capturing,  analyzing  and  assisting  the 
operator’s  information  handling  policy. 

The  experimental  studies  demonstrated  the  effectiveness  of  aiding  based 
on  the  time-continuous,  value-based  model  of  information  handling.  In 
terms  of  performance  score,  the  aiding  of  model  recommendation  andp 
parameter  aggregation  was  more  effective  than  the  aiding  of  parameter 
aggregation  alone,  and  the  improvement  over  unaided  situations  was  sub¬ 
stantial  in  both  high  and  low  rate  conditions.  In  support  of  this,  sub¬ 
jects  perceived  the  effectiveness  of  and  expressed  the  desirability  for 
the  aiding,  although  both  aiding  conditions  had  imposed  extra  loading 
over  the  subjects. 

In  general,  the  addition  of  model  recommendation  was  not  perceived  by 
the  subjects  as  desirable  in  the  aiding  of  data  aggregation  alone.  This 
was  due  to  the  frequent  requirements  of  operator  overrides  in  the  ini¬ 
tial  model -training  period.  It  is  possible,  however,  to  separate  the 
training  periods  from  the  experiment  sessions,  so  that  the  model  recom- 
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mendation  can  be  better  perceived  to  be  "in-tune"  with  the  subjects'  own 
preference.  On  the  other  hand,  increased  accuracy  of  model  recommenda¬ 
tion  is  also  possible.  The  probability  estimation  programs  were  frozen 
throughout  the  experimental  sessions  and  the  attribute  weights  were 
reassigned  to  be  unified  in  the  beginning  of  each  phase.  Dynamic  esti¬ 
mation  of  outcome  probabilities  specific  to  each  subject  should  improve 
the  accuracy  of  the  attribute  level  estimates  and,  in  turn,  result  in 
better  information  recommendations.  Similarly,  allowing  the  continued 
use  of  trained  weights  could  update  the  convergence  of  weight  vector, 
reduce  the  model  training  period,  and  provide  more  coherent  recommenda¬ 
tions. 


5.2  Management  of  Information  Handling 


The  availability  of  a  methodology  for  quantitative  representation  of 
both  descriptive  and  prescriptive  aspects  of  operator  information  han¬ 
dling  opens  up  the  possibility  of  managing  automated  sensor-based  sys¬ 
tems  through  supervisory  man-machine  systems;  where  event  sensing,  data 
aggregation,  source  selection  are  performed  by  the  machine  and  problem 
recognition,  event  selection,  and  consequence  evaluation  are  jointly 
performed  by  the  operator  and  the  adaptive  models.  Of  course,  the  adap¬ 
tive  modeling  approach  is  not  preferred  for  all  aspects  of  information 
management.  These  techniques  are  specific  to  the  tasks  that  require 
complex,  subjective,  recurrent,  and  stressed  decisions.  Therefore,  the 
approach  is  most  applicable  to  information  handling  that  features  some 
or  all  of  the  following  operational  characteristics: 

(1)  High  Information  Load.  The  operator  is  in  a  time-stressed 
decision  task.  For  each  decision  he  can  process  only  a 
portion  of  the  available  data  set  and  must  choose  an  action 
within  a  short  time. 
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(2)  Costly  Information  Transmission.  The  transmission  of  data 
to  the  operator  is  subject  to  cost,  risk  of  detection,  or 
limited  transmission  capabilities.  Immediately  valuable 
information  must  be  selected. 

(3)  Significant  Judgmental  Factors.  The  decision  maker  must 
consider  the  credibility  and  content  of  the  evidence  along 
with  the  probabilities  and  utilities  of  the  consequences 
associated  with  each  ensuing  action. 

(4)  Multiple  Competing  Information  Sources.  A  variety  of  dif¬ 
ferent  systems  or  sensors  must  be  monitored,  and  each  unat¬ 
tended  system  increases  in  uncertainty  over  time. 

frnong  the  examples  of  actual  military  decision  making  situations  that 
require  such  tasks  are:  (1)  supervisory  control  and  decision  making  in 
advanced  aircraft;  (2)  air  traffic  control;  (3)  remotely  piloted  vehicle 
guidance;  (4)  satellite  intel ligence  coordination;  (5)  supervision  of 
air,  ground,  or  sea  support  operations. 
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EXPERIMENTAL  APPARATUS  AND  SOFTWARE 

The  experimental  situation  is  generated  by  a  PDP-11/45  driven  multiple- 
display  system  which  included  a  Genesco  Model  GCT-3Q0  programmable  color 
grap  -  system  with  eight  color,  480-640  raster,  high  speed  full  graphic 
display  monitor.  Also  included  are  the  programmable  control  and  selection 
console,  consisting  of  special-purpose  keypad  and  X-Y  joystick  interface. 
The  individual  hardware  components  are  illustrated  as  follows: 

(1)  Computer:  A  DEC  PDP-11/45  with  UNIX  operating  system  is  used  to 
load  and  executes  the  software  programs. 

(2)  Graphic  System:  The  Genisco  GCT-3000  color  graphic  system  generates, 
updates  and  displays  the  scenario  in  real  time. 

(3)  Programmable  Control  Console:  This  unit,  containing  (a)  an  8-key 
programmable  control  panel  and  (b)  a  spring-centered  X-Y  joystick, 
is  a  microprocessor-based  intelligent  terminal,  which  polls  the 
subject's  inputs,  reflects  the  computer's  decision  and  converts 

the  analog  signal  of  the  joystick  (using  a  multiplexed  A/D  converter). 
The  unit  is  specifically  designed  to  support  the  experimental 
study.  It  provides  two-way  sampled  signal,  power,  clock  signal, 
logic-level  buffering  and  conversion.  This  is  shown  in  the  func¬ 
tional  block  diagram  of  Figure  1-1. 

(4)  Secondary  Display:  A  CRT  terminal  is  used  to  display  and  update 
the  subsystem  task  status  indicators  under  the  control  of  the 
computer,  and  reports  the  subject's  actions  back  to  the  computer. 


FIGURE  1-1. 

PROGRAMMABLE  CONTROL/DECISION  INTERFACE 
SCHEMATIC  ORGANIZATION 


The  software  was  developed  based  upon  the  structural  programing  technique 
of  the  C  language.  Figure  1-2  presents  a  flow  chart  of  the  main  control 
software,  and  the  organization  of  major  subroutines,  which  are  described 
as  follows: 

MAIN  --  This  routine  monitors  and  controls  the  overall  system  logic 
flow,  and  calls  all  subroutines  in  a  pre-defined  sequence.  The  real¬ 
time  process  queue  and  interrupt  structure  is  also  implemented  in  the 
MAIN  program. 

INITIALIZATION  --  This  program  handles  (a)  session  intialization  and 
(b)  phase  initialization. 

REPORT  —  Phase  and  session  reports  are  generated  and  stored  in  the  data 
file  for  the  off-line  data  analysis  and  reporting. 

SCENARIO  —  The  real-time  presentation  and  updates  of  the  scenario  are 
made  by  replacing  the  outgoing  portions  of  the  continuous  target 
paths/terrain  with  a  new  incoming  target  path/terrain  on  the  top  part 
of  the  display.  The  messages  are  also  updated  accordingly. 

MAU  --  The  MAU  model  generates  the  tate  and  the  information  probabilities, 
and  calculates  the  attributes,  iased  upon  the  highest  MAU,  the  model 
makes  both  messages  and  action  recommendations  for  the  next  move. 

ADJUSTMENT  —  The  utility  weights  are  adaptively  adjusted  based  upon  the 
subject's  decisions  and  actions. 
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EXPERIMENTAL  SITUATION  SOFTWARE  PROGRAM  ORGANIZATION 
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FIGURE  1-2.  (Continued) 
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The  flight  information  handling  simulation  and  model-based  aiding  algorithm 
plus  data  collection  routine,  running  under  UNIX  operating  system, 
requires  approximately  74K  bytes  of  core.  The  program  flow  including 
threat  assessment  and  computer  aiding  is  as  follows. 

First,  the  sytem  is  initialized.  The  experimenter  interactively 
specifies  the  task  conditions  and  experiment  identification  to  the 
computer.  The  appropriate  disk  file  (prepared  beforehand)  is  read  in, 
which  contains  all  the  information  needed  by  the  simulation  program  to 
generate  the  proper  target  type  display  and  subsystem  event  arrivals. 

(The  task  is  mainly  specified  by  the  contents  of  this  disk  file,  so  that 
experimental  conditions  can  easily  be  changed.)  The  data  file  in  which 
sampled  data  is  to  be  stored  is  opened  and  the  time  and  date  are  recorded. 

The  program  then  enters  the  main  iteration  loop  for  three  tactical  phase 
and  for  the  given  time  period.  The  program  then  first  updates  the 
attribute  weight  and  attribute  level  evaluation,  the  information  value 
and  provide  message  recommendation.  During  the  loop,  the  program  also 
waits  for  keyboard/control  response  from  the  subject.  For  example,  when 
a  response  is  made,  and  a  subsystem  selected  for  diagnostic  action,  one 
of  two  things  can  happen.  If  no  event  has  occurred  in  the  selected 
subsystem,  a  false  alarm  has  been  made,  and  the  loop  is  begun  again. 

If  an  event  has  occurred  in  the  sybsystem,  the  checklist  diagnostic 
procedure  is  begun.  If  the  subject  makes  an  incorrect  response  in  the 
checklist  procedure,  the  main  loop  is  immediately  restarted.  Otherwise, 
after  the  response  to  the  event  is  correctly  completed,  the  subsystem 
pointer  is  redrawn  (upward),  and  another  event  is  scheduled  for  that 
subsystem,  before  the  loop  is  restarted. 

As  the  main  iteration  loop  is  being  executed,  the  real-time  clock  is 
running,  and  checks  are  made  frequently  to  determine  if  it  is  time  to 
perform  a  system  states  and  ctatus  update.  Update  of  the  simulation 
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state  is  made  every  0.50  second.  Several  things  happen  during  a  simulation 
update.  First,  a  data  sample  is  taken,  and  stored  on  the  PDP-11  disk.  The 
states  of  target,  own  vehicle  and  the  status  of  each  subsystem,  control 
inputs  and  keyboard  responses,  and  the  states  of  the  own  vehicle  dynamics 
are  sampled.  Also  included  are  computer  aiding  status  in  the  aided  session. 

Next,  a  schedule  of  events  is  checked,  and  if  an  event  is  to  occur  at 
the  present  time  or  if  a  keyboard  input  is  received,  the  corresponding 
subsystem  indicator  is  redrawn  (downward)  on  the  display. 

The  threat  neutralization  procedure  is  then  performed.  The  positions  of 
the  targets  are  updated,  and  redrawn  on  the  display.  The  subject's 
action  decision  is  examined  and  outcome  determined,  and  displayed.  At 
this  point,  the  simulation  update  is  complete,  and  the  main  iteration  loop 
is  resumed. 

When  the  subject  has  flown  over  the  three-phase  mission,  the  simulation 
ends.  At  this  time ,  information  such  as  starting  and  finishing  time  of  the 
trial,  subject  name,  experiment  identification,  date,  and  subjective 
pilot  comments  are  recorded  in  the  data  file. 
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SUBJECTS'  INSTRUCTIONS 


This  experiment  is  part  of  a  program  of  continuing  research  at  Percep- 
tronics  in  human  decision  making  and  information  handling.  The  purpose 
of  this  particular  experiment  is  to  analyze  ways  in  which  a  human  operator 
handles  multiple  information  demands  and  how  a  computer  might  aid  a 
human  operator  in  operating  a  simulated  aircraft.  You  are  an  integral 
part  of  this  research  since  your  performance  provides  the  baseline  data 
for  predicting  overall  man-machine  system  performance  and  estimating  the 
effectiveness  of  various  aiding  techniques. 

Task 

The  primary  task  is  one  of  navigating  a  simulated  aircraft  through  a 
changing,  hazardouz  environment.  The  environment  contains  threats  of 
uncertain  form  and  extent.  At  each  threat,  you  must  either  take  an 
avoidance  action  or  an  agressive  action.  The  identity  and  location  of 
the  obstacles  can  be  determined  by  requesting  different  types  of  infor¬ 
mation.  The  different  types  of  information  vary  in  cost  and  content. 

The  secondary  task  is  to  monitor  a  subsystems  display  which  simulates 
status  indicators  of  aircraft  functions  and  interactive  check-list 
procedures  which  correct  malfunctions. 

Displays  and  Controls 

The  environment  and  the  vehicle  are  shown  as  in  a  moving-map  display 
where  incoming  threats  appear  at  random,  presented  with  symbol  "D"  at 
the  upper  edge  of  the  screen,  and  move  downward  at  a  constant  velocity. 

You  can  move  the  vehicle  symbol  horizontally  to  avoid  the  obstacles,  or 
to  fire  at  the  nearest  threat.  Your  range  of  fire  is  indicated  by  1/3  of 
dashes  (-)  placed  at  the  extreme  right  and  left  of  the  screen  (see  Figure  1). 
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FIGURE  II-l. 

MAP  SITUATION  ANO  OUTCOME  DISPLAY 


The  threats  introduce  uncertainty  to  the  task.  Each  type  of  obstacle 
has  a  probability  distribution  of  avoidance.  The  first  obstacle  is  a 
helicopter  (H).  If  you  pass  close  to  it  there  is  a  high  probability  of 
damage.  The  second  threat  is  an  aircraft  (A),  which  has  a  wide  area  of 
danger.  Again,  a  pass  close  to  it  should  be  avoided.  The  third  threat 
is  a  missile  (M),  which  is  safe  close  to  it  but  not  safe  far  away. 

Often,  it  is  best  to  fly  through  or  around  it.  The  fourth  symbol,  a 
bird  (B),  is  a  non-dangerous  object.  It  cannot  damage  you  and  cannot 
be  shot  down  (see  Figure  2). 

After  reviewing  the  mission  situation  display,  and  before  selecting 
information,  you  may  decide  on  taking  one  of  the  following  actions: 

(1)  Avoidance  -  try  to  avoid  the  damage;  the  results  will  be 
'avoid1  or  'damage'. 

(2)  Aggression  -  try  to  hit;  the  results  will  be  'hit  and 
avoid',  'miss  and  avoid',  and  'miss  and  damage'. 

If  no  action  is  taken,  avoidance  is  assumed.  The  amount  of  gain  and  loss 
(and  thus  the  score)  depend  on  the  information  costs  and  payoffs  for 
Successful /Unsuccessful  Avoidance  and  Successful /Unsuccessful  Aggression. 
Payoffs  will  vary  from  phase  to  phase. 

The  task  consists  of  a  series  of  three  phases  of  aircraft  missions: 
cruise,  surveillance,  and  aggression.  Threat  probability  and  payoffs 
will  vary  among  phases.  For  example,  there  is  a  high  probability  of  a 
bird  in  the  cruise  phase  and  a  high  probability  of  a  missile  in  the 
aggression  phase.  To  make  an  information  selection,  you  can  press  one 
of  the  buttons  on  the  control  panel.  If  'full'  information  is  selected, 
the  differentiated  symbols  in  their  proper  location  will  move  down  the 
screen.  If  "outline"  information  is  selected,  the  missile  and  bird 
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H-Hel i copter 
B-BIrd 


CORRIDOR  LOCATION 


FIGURE  II-2. 

THREAT  CHARACTERISTICS 


symbols  will  be  differentiated,  but  helicopters  and  airplanes  will  be 
represented  by  a  single,  non-differentiating  symbol  (A).  Similarly, 
"biological"  information  will  use  a  single,  non-differentiated  symbol, 
(MAH)  to  represent  either  missile,  airplane,  or  helicopter.  "Location" 
information  provides  no  discrimination.  A  symbol  (X)  denotes  the 
location  but  not  the  identity  of  threats.  "Threat"  information  discri¬ 
minates  the  obstacles  using  the  standard  symbols,  but  locates  them 
only  as  lying  in  the  left  or  fight  half  of  the  screen.  "Default"  infor¬ 
mation  gives  no  target  identity  nor  target  location  (see  Table  1). 

Situational  Display.  The  stages  of  an  aircraft  mission  can  be  charac¬ 
terized  by  such  factors  as  danger,  difficulty,  system  reliability,  and 
communications  security.  The  situational  conditions  that  vary  among 
phases  are: 

(1)  Degree  of  danger  -  this  is  the  distribution  of  possible 
threats  in  a  given  phase.  A  different  set  of  probabilities 
of  the  4  threat  types  is  assigned  to  each  phase. 

(2)  Information  accuracy  -  a  percentage  of  information 
transmissions  are  inaccurate  through  having  random 
nunbers  added  to  location  content.  This  indicates  the 
possibility  of  the  location  being  unpredictable. 

(3)  Payoffs  -  different  payoffs  in  points  are  made  for 
avoidance  of  or  damage  sustained  from  the  threats,  and/ 
or  for  successful  or  unsuccessful  aggression  actions 
toward  the  threats.  Each  of  the  payoffs  will  vary  phase- 
by-phase.  Your  payoff  for  successful  and  unsuccessful 
actions  are  presented  by  two  numbers  with  a  slash  between 
them,  for  example:  12/6.  The  first  number  adds  to  your 
score  (if  successful)  while  the  second  number  subtracts 
from  your  score  (if  unsuccessful). 


TABLE  II-l 


INFORMATION  SOURCE  CHARACTERISTICS 


Information  Type 

Discrimination 

Location 

Symbol s 

1.  Full 

All 

All 

M,  A,  H,  B 

2.  Outline 

All  but  Airplane/ 
He! i copter 

All 

M,  AH,  B 

3.  Biological 

Bird  Only 

All 

MAH,  B 

4.  Location 

None 

All 

X 

5.  Left/Right 

All 

Left/Right 

M,  A,  H,  B 

6.  Default 

None 

None 

D 
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The  situational  conditions  that  vary  within  a  phase  are: 


(4)  Costs  -  a  different  cost  is  assigned  to  each  information 
choice.  This  is  the  number  of  points  that  the  infor¬ 
mation  costs. 

(5)  Delay  -  the  delay  in  half  seconds  before  actual  display 
of  the  selected  information.  This  is  equivalent  to  the 
portion  of  the  distance  the  default  symbol  travels  before 
the  "true"  symbol  is  displayed. 

(6)  Detection  -  the  increased  danger  on  the  succeeding  decision 
due  to  the  use  of  a  given  information  source.  A  .10  de¬ 
tection  means  that  a  damage  will  increase  by  10%  on  next 
decision. 

Your  score  is  updated  based  on  the  payoff  and  cost  incurred  in  your 
decision,  i.e..  Added  Score  =  Payoffs  -  Cost.  For  example,  in  Figure  1, 
if  the  outcome  is  Miss  and  Avoid  by  using  Full  information,  the  added 
score  is  -8  +  12  -  10  =  -6.  Therefore,  sound  selection  and  tradeoffs 
between  potential  payoffs,  cost,  delay,  accuracy  and  detection  are 
required  to  achieve  high  score.  In  some  sessions,  computed  recommendation 
on  selection  will  be  provided.  You  may  accept  or  override  the  recoirmenda- 
tion  by  pressing  the  buttons,  otherwise,  the  default  information  and/or 
the  recommended  action  will  be  taken  by  the  system. 

Subsystem  Tasks.  The  subsystem  task  requires  the  operator  to  monitor 
the  top-level  subsystem  processes  to  detect  possible  events  (i.e.,  "0" 
symbols)  and  to  act  to  bring  the  process  into  operational  status  (i.e., 

"1"  symbols).  If  the  operator  thinks  process  6  is  down,  he  may  press 
"6"  on  the  keypad  and  the  display  of  next  level  appears.  The  display 
may  then  show  that  branch  3  is  "down"  and  a  press  of  key  "3"  will  lead 
to  the  display  of  next  level,  where  the  branch  "4"  is  "down,"  etc.  He 
continues  until  all  branches  are  "up"  whereupon  process  6  is  "up"  again. 
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Speed  and  errors  will  be  recorded  on  performance  data.  When  the  incorrect 
key  is  pressed  an  error  will  be  registered  in  the  system  while  the  display 
status  remains  unchanged.  Remember,  it  is  assumed  that  the  subsystem  1 
has  the  highest  priority  and  subsystem  6  has  the  lowest.  Your  overall 
performance  will  be  evaluated  based  on  the  total  score  achieved  in  the 
primary  task  as  well  as  the  speed  and  errors  in  the  secondary  tasks. 
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APPENDIX  III 

SUBJECTIVE  RATING  QUESTIONNAIRE 


APPENDIX  III 

SUBJECTIVE  RATING  QUESTIONNAIRE 


At  the  end  of  each  experimental  run,  an  appropriate  version  of  rating 
questionnaire  shown  in  Figure  III-l  was  giv  to  the  subject.  On  the 
rating  scale  following  each  question,  subjects  made  a  mark  indicating 
their  perception  of  relative  effort  and  quality  of  computer  aiding.  These 
ratings  were  then  quantified  and  scaled  for  statistical  analysis.  Subject's 
comments  were  also  summarized. 
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Below  you  will  find  several  questions  pertaining  to  the  computer  aided 
task  you  have  just  completed.  Underneath  each  question  is  a  line  with 
5  classifications.  Please  place  a  slash  at  the  point  you  feel  best 
classifies  the  question.  It  is  not  necessary  to  mark  directly  at  a 
classification  point  unless  you  feel  it  is  appropriate.  For  example: 

How  comfortable  was  the  temperature  of  the  room? 


very cool comfortable 7 warm very 

warm 

Her  the  slash  is  placed  between  comfortable  and  warm,  indicating  that  the 
room  temperature  was  approximately  halfway  between  comfortable  and  warm. 


If  you  do  not  understand  the  instructions,  please  ask  the  experimenter  to 
clarify  them.  If  you  do  not  have  any  questions  then  please  begin: 


1.  What  effect  do  you  think  computer  recommendation  had  in  the  overall 
performance? 

large  modest  slight  no  slight  modest  Targe 

degradation  degradation  degradation  effect  improvement  improvement  improvement 


2.  How  desirable  did  you  find  the  computer  recommendation? 

definitely  somewhat  doesn't  somewhat  def initely 

don't  like  it  undesirable  matter  desirable  desirable 


3.  How  easy  was  it  to  use  the  computer  recommendation? 

very  difficult  reasonable  easy  very 

difficult  easy 


4.  Did  you  feel  the  recommendations  gave  you  a  sense  of  control  in 
handling  the  situations? 

not  at  all  somewhat  moderate  more  than  very 

usual  much 


5.  How  satisfied  were  you  with  the  quality  of  the  information  recommendation? 


not  at  all  somewhat  moderate  more  than  very 

usual  much 


FIGURE  III-l. 

SUBJECTIVE  RATING  QUESTIONNAIRE 
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6.  What  level  of  effort  and/or  attention  did  you  have  to  expend? 

very  Tow  moderate  high  very 

low  high 


7.  How  complicated  or  difficult  is  the  task? 

extremely  simple  moderate 

simple 


complicated  extremely 

complicated 


8.  General  observations  and/or  additional  comments. 


FIGURE  III-l.  (Continued) 
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APPENDIX  IV 
ANALYSIS  OF  VARIANCE 


Two  sets  of  analyses  of  variance  based  on  the  data  for  twelve  subjects 
was  conducted.  The  first  set  consists  of  repeated-measures  ANOVA  for 
task  performance  measures  and  subjective  ratings,  to  be  presented  in 
Table  IV-1  through  IV-7.  The  second  set  consists  of  the  MANOVA  on  the 
attribute  weight  vector.  The  results  of  the  analysis  are  presented  in 
Table  IV-8  with  the  statistics  for  each  effect  given  separately. 
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TABLE  IV-1 

ANOVA  ON  PERFORMANCE  SCORE 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 

50753728.00 

1 

50753728.00 

702.57 

0.000 

ERROR 

794637.12 

11 

72239.68 

M  (MODE) 

960870.00 

2 

480435.00 

112.10 

0.000* 

ERROR 

94284.00 

22 

4285.63 

S  (SPEED) 

5116385.00 

1 

5117385.00 

473.04 

0.000* 

ERROR 

118999.62 

11 

10818.14 

MS 

96370.62 

2 

48185.31 

16.96 

0.000* 

ERROR 

62507.58 

22 

2841.25 

SOURCE 

TABLE  IV-2 

ANOVA  ON  SUBSYSTEM  WAITING  TIME 

SUM  OF  DEGREES  OF  MEAN  F 

SQUARES  FREEDOM  SQUARE 

TAIL 

PROBABILITY 

MEAN 

30243.55 

1 

30243.55 

31.90 

0.000 

ERROR 

10427.66 

11 

947.96 

M  (MODE) 

11.01 

2 

5.50 

0.31 

0.733 

ERROR 

384.75 

22 

17.48 

S  (SPEED) 

1397.17 

1 

1397.17 

9.51 

0.010* 

ERROR 

1616.89 

11 

146.99 

MS 

77.81 

2 

38.90 

0.77 

0.476 

ERROR 

1115.91 

22 

50.72 
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TABLE  IV-3 


ANOVA  ON  SUBJECTIVE  RATINGS  OF  EFFORT 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 

4948.45 

1 

4948.45 

212.52 

0.000 

ERROR 

256.12 

11 

23.28 

M  (MODE) 

28.41 

2 

14.20 

2.05 

0.153 

ERROR 

152.53 

22 

6.93 

S  (SPEED) 

79.48 

1 

79.58 

20.46 

0.001* 

ERROR 

42.78 

11 

3.88 

MS 

0.75 

2 

0.37 

0.12 

0.884 

ERROR 

66.85 

22 

3.03 

SOURCE 

TABLE  IV-4 

ANOVA  ON  SUBJECTIVE  RATINGS  FOR  THE 

EASE  OF  USE  OF  COMPUTER  AIDING 

SUM  OF  DEGREES  OF  MEAN  F 

SQUARES  FREEDOM  SQUARE 

TAIL 

PROBABILITY 

MEAN 

5557.73 

1 

5557.73 

488.29 

0.00 

ERROR 

125.20 

11 

11.38 

M  (MODE) 

0.00 

1 

0.00 

0.00 

0.986 

ERROR 

63.60 

11 

5.78 

S  (SPEED) 

12.91 

1 

12.91 

8.83 

0.013* 

ERROR 

16.09 

11 

1.46 

MS 

2.66 

1 

2.66 

1.16 

0.305 

ERROR 

25.23 

11 

2.29 
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TABLE  IV-5 


ANOVA  ON  SUBJECTIVE  RATINGS  OF 
DESIRABILITY  OF  COMPUTER  AIDING 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 

5135.66 

1 

5135.66 

287.08 

0.000 

ERROR 

196.78 

11 

17.88 

M  (MODE) 

216.32 

1 

216.32 

56.57 

0.000* 

ERROR 

42.06 

11 

3.82 

S  (SPEED) 

11.11 

1 

11.11 

2.09 

0.176 

ERROR 

58.46 

11 

5.31 

MS 

0.04 

1 

0.04 

0.02 

0.897 

ERROR 

29.30 

11 

2.66 

SOURCE 

TABLE  IV-6 

ANOVA  ON  SUBJECTIVE  RATINGS  OF  AIDING  EFFECTIVENESS 

SUM  OF  DEGREES  OF  MEAN  F  TAIL 

SQUARES  FREEDOM  SQUARE  PROBABILITY 

MEAN 

4681.87 

1 

4681.87 

129.90 

0.000 

ERROR 

396.46 

11 

36.04 

M  (MODE) 

279.51 

2 

139.75 

5.27 

0.014* 

ERROR 

583.67 

22 

26.53 

S  (SPEED) 

1.62 

1 

1.62 

0.21 

0.654 

ERROR 

84.15 

11 

7.65 

MS 

15.86 

2 

7.93 

1.08 

0.357 

ERROR 

161.63 

22 

7.34 
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TABLE  IV-7 


ANOVA  ON  SUBJECTIVE  RATINGS 
OF  TASK  DIFFICULTY 


SOURCE 

SUM  OF 
SQUARES 

DEGREES  OF 
FREEDOM 

MEAN 

SQUARE 

F 

TAIL 

PROBABILITY 

MEAN 

4054.49 

1 

4054.49 

160.22 

0.00 

ERROR 

278.35 

11 

25.30 

M  (MODE) 

12.78 

2 

6.39 

1.32 

0.289 

ERROR 

106.89 

22 

S  (SPEED) 

38.57 

1 

38.57 

17.13 

0.002* 

ERROR 

24.76 

11 

2.25 

MS 

4.82 

2 

2.41 

2.01 

0.157 

ERROR 

26.36 

22 

1.19 

TABLE  IV-8 

MANOVA  ON  THE  ATTRIBUTE  WEIGHT  VECTOR 


N  A.L_Y_S.-I.S  OF  V  UJ.IU.C-E.M  *  *  *  »  »- 


EFFECT  ..  m  »?EE0  BY  PHASE 
"nULTIVARIATE  TESfS'  OF  SIGNIFICANCE"?*  =  4.  M_  =  0 H  »'  96) 


TEST  NAME 

VALUE 

APPROX.  F 

HYPOTH.  OF 

ERROR  OF 

SIG.  OF  F 

FILLAIS 

NOTELLINGS 

MILKS 

ROYS 

.17764 

.19565 

.83002 

.11823 

1.83112 

1.88315 

1.86158 

20.00 

20.00 

20 . 00 

788.00 

770.00 

644.38 

.913 

.911 

.913 

TJNIVARIATE'F^TESTSTIITH  (4,198) 

_or>. 

VARIABLE 

HYFOTH.  SS 

ERROR  SS 

HYPOTH.  MS 

ERROR  MS 

F 

SIG.  OF  F 

ATTRIS1 

.14153 

"  1.29318 

.  04038 

.00653 

6.18290 

.900 

ATTRIBZ 

.12616 

2.09861 

.03154 

.01060 

2.97610 

.020 

ATTRIBJ 

.16350 

6.30078 

.  04087 

.03182 

1.28445 

:277 

ATTRIB6 

.03016 

4.23385 

.00754 

.02138 

.  35266 

.842 

ATTRIB3  . 

. 04479 

3.80107 

.01120 

.01920 

.58322 

.475 

EFFECT  ..  SPEED  BY  PHASE 

MULTIVARIATE  TESTS  OF  SIGNIFICANCE  (1  >  t,M  1,  N  :  «) 


TEST  NAME" 

VALUE 

APPROX.  F 

HYPOTH.  OF 

ERROR  DP 

SIG?  OF  F" 

FILLAIS 

.32126 

7.46329 

10.00 

390.00 

0.0 

_  HOTELLINOS. 

.41116 

7.93534 

10.00 

386.00 

-0.0 

MILKS 

.49624 

7.49993 

10.00 

388.00 

0.0 

ROYS 

.25177 

UNIVARIATE 

F-TESTS  WITH  (2.198) 

b.  p. 

"variable 

1  HYPOTH.  SS 

ERROR  SS 

HYPOTH.  MS 

ERROR  NS 

F 

SIG.  OF  F 

ATTRIB1 

.10204 

1.29318 

.05102 

.00653 

7.81140 

.001 

ATTRIB2 

.20815 

2.09841 

.10407 

.01060 

9.82001 

.  000 

ATTRI83 

.19791 

6.30078 

.09896 

.03182 

.10970 

.047 

ATTRI84 

.18731 

4.23385 

.09365 

.02138 

4.37983 

.014 

ATTRI83 

.12107 

3.80107 

.06053 

.01920 

3.15324 

.045 

^FFECT  ..  MODE  »Y  FHASE 

MULTIVARIATEtESfrOF- SIGNIFICANCE'S  *  6.  ifi'oTN  =  96)' 


TEST  NAME 

VALUE 

APPROX.  F 

HYPOTH.  DF 

ERROR  DF 

SIG.  OF  F 

0.0 

0.0 

0.0 

FILLAIS 

HOTELLINOS 

MILKS 

.39758 

.49537 

.64226 

23007 

MU 

uoi> 

-flUH 

20.00 

20.00 

20.00 

788.00 

770.00 

644.38 

UNIVARIATE 

F=TESir"UlTH"(  4  7l  98 )  "O' 

,  F. 

VARIABLE 

HYPOTH.  SS 

ERROR  SS 

HYPOTH.  MS 

ERROR  NS 

F 

$10. 

OF  F 

'ATTRXBX 

27546 

— 1729318 

.06887 

.00453 

10.34407 

.  000 

ATTRIB2 

.21962  . 

2.09841 

.05490 

.01060 

3 . 18065 

.001 

ATTRIB3 

.73556 

6.30078 

.  18389 

.03182 

5.77870 

.  000 

ATTRIB4 

.59242 

4.23385 

.14810  .  _ 

.02138 

6 . 92622  _ 

- 

_t000 

ATTRIB5 

.23068 

— 3.80107 

.05767 

.01920 

3.00406 

.  -  .  • 

.  020 
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TABLE  IV-8  (Continued) 


EFFECT  ..  MODE  BY  PHASE  (CONT.) 

EFFECT  ".'PHASE  ”  '  ' 

MULTIVARIATE  TESTS  OF  SIOHIFICAHCE  (S  3  2.  M  3  1.  H  3  96 J 

TEST  NAME - VALUE  APPROX.”  F  HYPOTH.  OF  ERROR- OF  'SIQ.^OF  F 


PILLAIS 

HOTELLING!. 

MILKS 

ROYS 


1 .03531  41.SJ449 

2,23513 _ 43.13802 

'  .22778  42.49634 

.58747 


10.00 

10.00 

10.00 


390.00 

.386.00 

388.00 


0.0 

.0.0 

0.0 


univariate  f 

-TESTS  WITH  (2.198) 

0.  F. 

"variable 

ATTRIB1 

ATTRIB2 

flYPOTH.  SS 

.76778 

.68347 

ERROR- SS 

1.29318 

2.09841 

HYPOTH.  MS 

.38389 

.34174 

ERROR 'MS 

.00653 

.01060 

F 

38.77794 

32.24532 

sTo.  o  f  p— 

0.0 

0.0 

ATTRI83 

.59460  . 

6”.  30078 

.29730 

.03182 

9.34261 

.000 

ATTRI84 

3.47722 

4.23385 

1.73861 

.02138 

81.30782 

0.0 

ATTRIB5 

1.74443 

3.80107 

.87221 

.01920 

45.43411 

0.0 

EFFECT  ..  MODE 


TEST  NAME 


PILLAIS 

HOTELLIHOS 

MILKS 

_ROYS _ 


VALUE 


.79390 
3.64301 
.21373 
_ .786i.7_ 


CS  3  2.  M  3  1. 

APPROX.  F 

N  3  96) 

HYPOTH.  OF 

ERROR  OF 

SIO.  OF 

25.67132 

10.00 

390.00 

0.0 

70.31009 

10.00 

386 .00 

0 . 0 

45.12584 

10.00 

388.00 

0 . 0 

TINIVARIATE  F-TESTS”TlITH~(27l98>'  O.  F. 
VARIABLE  HYPOTH.  SS  ERROR  SS 


HYPOTH,  MS 


"ATTRIB1 

ATTRIB2 

ATTRI83 

ATTRI84 

'ATTRI8S” 


3767601 
.00327 
1.11787 
.34843  _ 


.89842 


1.29318 

2.09841 

6.30078 

4.23385 

"3780107” 


1.83801 
.  00263 
.53894 
.17422 
.44921 


ERROR  MS 

””  "”.00653- 
.01060 
.03182 
.02138 
'  701920 


SIO.  OF  F 


281.41810 
*  .24852 

17.56435 
8.14775 
23.39952 


0.0 

.780 

.000 

.000 

o'.o 


EFFECT  ■■  SPEED 

MULTIVARIATE-tESTS-^F'SIONIFICAHCE  (S  3  1,  M~*”l  '1727V»"mT 


TEST  NAME 

VALUE 

APPROX.  F 

PILLAIS 

.36713 

22.50812 

HOTELLINOS 

.58011 

22.50812 

MILKS 

.63287 

22.50812 

ROYS 

.36713 

UNIVARIATE 

• 

F-TESTS  WITH  (1.198) 

0.  F. 

VARIABLE 

HYPOTH. "SS 

ERROR  SS 

ATTRIB1 

.30827 

1.29318 

ATTRIB2 

.65230 

2.09841 

ATTRIB3 

.43023 

6.30078 

ATTRIB4 

.00167 

4.23385 

ATTRI8S 

.88039 

3.80107 

HYPOTH.  OP 

ERROR  OF 

SIO.  OF 

3.00 

194700 

0.0 

5.00 

194,00 

00 

5.00 

194.00 

0.0 

HYPOTH.  MS 

ERROR  MS 

F 

.30827 

.00653 

47.19888 

.65230 

.01060 

61.54927 

.43023 

.03182 

13.51982 

. 00167 

.02138 

.07794 

.68039 

.01920 

45.85995 

SI 6.  OF  F 


O.o 
0.0 
■  OM 
.780 
0.0 
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